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(54) - Video ^coding and decoding apparatus 

(57) An encoding apparatus includes an encoder for 
encoding an alpha-map signal for discriminating a back- 
ground from an object of an input picture in motioaebm- 
pensation prediction (MV) ^ trahsfofrn encoding Which 
uses -MV in a domain of each of N X N transform coef- 
ficients (n), a ,trarisfortn;circujt (101) for transforming Pf 
into n in aasordance with the alpha^nap signal, an in- 
verse transform ci rcuit for reconstructing pf by inversely 
transform ingn in accordance with the alpha-map signal, 
a selector (300) for obtaining a motion compensation 
predictkxi value (p) in the mth layer (m = 2 to M) by 
switching p in the mth layer and £ in the (m-1)th layer 
for each n, the selector (300)selectingf> in the mth layer 
for n by which a quantized output (Q) in the (m : 1)th.layer 
is 0 and selecting £,in the (m-1 )th layer for n by which 
Q = 1 -or. more, an adder ( il 0) f or calculating a difference 
df between a prediction error signal in the mth layer and 
a dequantized output in the (m-l)th layer, and an encod- 
er (140) for encoding and outputting the quantized sig- 
nal of df . This encoding apparatus realizes SNR scala- 
bility in M layers. 
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The pressor invention reiates to video encoding and decoding apparatuses for encoding a picture s.gnal at a high 
efficiency and transmitting or storing the encoded signal and, more particularly, to video encoding and decoding ap- 
paratuses with a scalable f unction capable of scalable coding by which the resolution and the image quality can be 

changed into multiple, layers. ^ . „„ 

Generally a picture signal is compression-encoded before being transmitted or stored because the signal has an 
enormous amount of information. To encode a picture signal at a high efficiency, pictures whose unit » a frame are 
divided into a plurality of blocks in units of a predetermined number of pixels. Orthogonal transform is performed for 
each block to.separate the spacial frequency of a picture into frequency components. Each frequency component is 

obtained as a transform coefficient and encoded. . • . • 

As one function of video encoding, a scalability function is demanded by which the image quality (SNR: Signal to 
Noise Ratio), the spacial resolution, and the time resolution can be changed step by step by .parts* decoding a bit 

Stf<? Tne scalability function is incorporated into Video Part (IS1 3816-2) of MPEG2 which is standardized in ISCVIEC. 

This scalability is realized by hierarchical encoding methdds: The scalability includes an encoder and a decoder 
of SNrtscalabil^ 

: in the encoder, layers are divided into a base layer (lower layer) whose image quality is low and an enhancement 
layer (upper layer) whose image quality is high. 

In the base layer, data is encoded by MPEG 1 or MPEG2: In the enhancement layer, the data encoded by the base 
laver is reoDnstructed and the reconstructed base layer data is subtracted from the enhancement layer data Only the 
resulting error is quantized by a quantization step size smaller than the quantization step size in the base layer and 
encoded That is the data is more finely quantized and encoded. The resolution can be increased by adding the 
er^hancement layer information to the base layer information, and this makes the transmission and storage of hkjh- 

25 quality pictures feasible. . - K „ , h . 

As described above, pictures are divided into the base layer and the enhancement layer, data encoded by the 
base layer is reconstructed, the reconstructed data is subtracted from the original data, and only the resulting error is 
qiiantizedijya quantization step size smaller than the quantization step size in the base layer and encoded. Conse- 
quently pictured can be encoded and decoded at a high resolution, This^echnique is called SNR scatabdrty^ 

k tte encoder an input picture is supplied to the base layer and the enhancement layer. In the base layer, the 
input picture is so processed as to obtain an error from a motion compensation- prediction rvalue obtained from apjciure 
c4 theWviouS f ian>e. and the error is subjected to orthogonal transform (OCT). The transform coefficient is quantized 
and variable-length^decocied to obtain a base layer output The c^ntized output is dequantized, subjected to awerse 
DCTaYrtacd^^ 

Motion compensation prediction is performed. on the basis of this frarne picture to obtain the motion compensation 

prediction value Lot the previous frame. - ' '. ; " : ' . • 

In the enhancement layer, on the other hand, the input picture is delayed until the predawn value is obtained from 
the base layer and processing is performed to obtain an error from a motion compensation prediction value m the 
enhahcernent layer obtained from the picture of the previous frame: The error is then subjected to orthogonal transform 
(OCT) arid the transform coefficient is corrected by using trie dequantized output from the base layer, quantized, and 
variabie-length^decoded. thereby obtaining an enhancement jayer output; The quantized ^ 
with the motion compensation prediction value of. the previous frame obtained in the base layer, and subjected to 
inverse D^T' A-frame picture is obtained by adding to the resuHdf the inverse OCT the motion compensation prediction 
value of the previous frame obtained in the enhancement layer. Motion compensation prediction is performed on the 
basis of this frame picture to obtain a motion compensation prediction value of the previous frame in the enhancement 

layS [n mis way video pictures can be encoded by using the SNR scalability: Note that although this SNR scalability is 
expressed by two layers, various SNR reconstructed pictures can be obtained by increasing the number of layers. 

In the decoder the variable-length decoded data of the enhancernent layer and the variable-length encoded data 
of the ba^' ^Wilcn^sepafateV'-^MPpiWd are separately variafcle-tength-decoded and oec^iar^ed. The two 
dequantized d^te are added, and the result is subjected to inverse OCT. The picture signal is restored by adding the 
motion compensation prediction value of the previous frame to the resultof the inverse DCT. Also, motion compensation 
. ptedk^ ^porfonned an the basis of a picture in an immediately previous frame obtained from the restored picture 
^ signal, thereby obtaining a motion compensation prediction value of the previous frame. 
55 The foregoing are examples of encoding and decoding using the SNR scalability. 

On the other hand, the spacial scalability is done on the basis of the spacial resolution, and encoding is separately 
performed in a base layer whose spacial resolution is low and an enhancement layer whose spacel resolution is high, 
in the base layer encoding is performed by Using a normal MPEG2 encoding method. In the enhancement layer, up- 
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sampling (in which a high-resolution picture is formed by adding pixels such as average values between pixels of a 
low-resolution picture) is performed for the picture from the base layer 1o thereby form a picture having the same si7e 
as the enhancement layer Prediction is adaptivety performed on the basis of motion compensation prediction using 
the picture of the enhancement layer and motion compensation prediction using the up-sampled picture. Consequently. 
5 encoding can be performed at a high efficiency 

The spacial scalability exists in order to achieve backward compatibility by which, for example, a portion of a bit 
stream of MPEG2 can be extracted and decoded by MPEG1 . That is. the spacial scalability is not a function capable 
of reconstructing pictures with various resolutions (reference: "Special Edition MPEG", Television Magazine, Vol. 49. 
No. 4, pp. 458-463, 1995). 

io More specifically, the video encoding technology of MPEG2 aims to accomplish high -efficiency encoding of high- 

quality pictures and high-quality reconstruction of the encoded pictures. In this technology, pictures faithful to encoded 
pictures can be reconstructed. 

Unfortunately, with the spread of multimedia, there is a demand for a reconstructing apparatus capable of fully 
decoding data of high-quality pictures encoded at a high efficiency, as a system on the reconstruction side. In addition, 

is there are demands for a system such as a portable system which is only required to reconstruct pictures regardless 
of whether the image quality is high, and for a simplified system by which the system price is decreased. 

To meet these demands, a picture is divided into, e.g., 8x8 pixel matrix blocks and DCT is performed in units of 
blocks. In this case, 8x8 transform coefficients are obtained. Although it is originally necessary to decode the data 
from the first low frequency component to the eighth low frequency component, the data is decoded from the first low 

20 frequency component to the fourth low frequency component or from the first tow frequency component to the sixth 
low frequency component. In this manner decoding is simplified by restoring the picture by reconstructing the signal 
of 4 x 4 resolution or the signal of 6 x 6 resolution, rather than the signal of 8 x 8 resolution. 

Unfortunately, when a picture which originally has 8x8 information is restored by using 4 x 4 or 6x6 information, 
a mismatch occurs between the restored value and the motion compensation prediction value, and errors are accu- 

25 mulated. This significantly degrades the picture. Therefore, it is an important subject to overcome this mismatch be- 
tween the encoding side and the decoding side. 

Note that as a method of converting the spacial resolution in order to control the difference between the spacial 
resolutions on the encoding side and the decoding side, there is another method, although the method is not stand- 
ardized, by which the spacial resolution is made variable by inversely converting some coefficients of orthogonal trans- 

30 form (e g., DCT (Discrete Cosine Transform)) by an order smaller than the original order. 

...... Unfortunately, when motic>ncx>mpensatkDn prediction is performed by image 

quality degradation caHed^a drift resulting from the. motion c*>npensation pretfction occurs in the reconstructed picture 
(reference: Iwahashi et al., 'Motion Compensation for Reducing Drift in Scalable Decoder*, Shingaku Giho IE94-97, 
1994). 

35 Accordingly, the method has a problem as a technique to overcome the mismatch between the encoding side and 

the decoding side 

On the other hand, the spacial scalability exists in order to achieve backward compatibility by which, for example, 
a portion of a bit stream of MPEG2 can be extracted and decoded by MPEG1 . That is, the spacial scalability is not a 
function of capable of reconstructing pictures with various resolutions (reference: 'Special Edition MPEG", Television 
*o Magazine, Vol. 49, No. 4, pp. 458-463, 1 995). Since hierarchical encoding is performed to realize the scalability function 
as described above, information is divisionally encoded and this decreases the coding efficiency. 

A video encoding system belonging to a category called mid-level encoding is proposed in "J.Y.A. Wang et. al., 
■Applying Mid-level Vision Techniques for Video Data Compression and Manipulation", M.I.T. Media Lab. Tech. Report 
No. 263, Feb. 1994". 

45 in this system, a background and an object are separately encoded. To separately encode the background and 

the object, an alpha-map signal which represents the shape of the object and the position of the object in a frame is 

necessary. An alpha-map signal of the background can be uniquely obtained from the alpha-map signal of the object. 
In an encoding system like this, a picture with an arbitrary shape must be encoded. As a method of encoding an 

arbitrary-shape picture, there is an arbitrary-shape picture signal orthogonal transform method described in previously 
so filed Japanese Patent Application No. 7-97073. In this orthogonal transform method, the values of pixels contained in 

a specific domain are separated from an input edge block signal by a separation circuit (SEP), and an average value 

calculation circuit (AVE) calculates an average value a of the separated pixel values. 

If an alpha-map indicates a pixel in the specific domain, a selector (SEL) outputs the pixel value in the specific 

domain stored in a block memory (MEM). If the alpha-map indicates another pixel, the selector outputs the average 
55 value a. The block signal thus processed is subjected to two-dimensional DCT to obtain transform coefficients for pixels 

in the specific domain. 

On the other hand, inverse transform is accomplished by separating the pixel values in the specific domain from 
pixel values in the block obtained by performing inverse DCT for the transform coefficient. 
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As described above, in the scalable encoding method capable of dividing pictures into multiple layers ; the coding 
efficiency is sometimes greatly decreased when video pictures are encoded In addition, scalable encoding by which 
the resolution and the image quality can be made variable is also required in an arbitrary-shape picture encoding 
apparatus which separately encodes the background and the object. It is also necessary to improve the efficiency of 

5 motion compensation prediction encoding for an arbitrary-shape picture 

On the other hand, the mid-level encoding system has the advantage that a method of evenly arranging the internal 
averagevalue of the object in the background can be realized with a few calculations. However, a step of pixel values 
is sometimes formed in the boundary between the object and the background. If DCT is performed in a case like this, 
a large quantity of high-frequency components are generated and so the amount of codes is not decreased. 

10 It is an object of the present invention to provide an encoding apparatus and a decoding apparatus capable of 

improving the coding efficiency when video pictures are encoded by a scalable encoding method by which pictures 
can be divided into multiple layers 

It is another object of the present invention to provide a scalable encoding apparatus and a scalable decoding 
apparatus capable of making the resolution arid foe image quality variable and improving the coding efficiency in an 

is arbitrary-shape picture encoding apparatus which separately encodes a background and an object. 

It rs still another object of the present invention to improve the efficiency of motion compensation prediction en- 
coding for arbrtrary-shape pictures. 

It is still another object of the present invention to alleviate the drawback that the code amount is not decreased 
due to the generation of a large quantity of high-frequency components when DCT is performed, even If a step of pixel 

20 values is formed in the boundary between an object and a background when a method of evenly arranging an internal 
average value of the object in the background is used. 

According to the present invention, there is provided a video encoding apparatus comprising: an orthogonal trans- 
form circuit for orthogonally transforming an input picture signal to obtain a plurality of transform coefficients; a first 
local decoder for outputting first transform coefficients for a fine motion compensation prediction picture on the basts 

25 of a previous picture; a second local decoder for outputting second transform coefficients for a coarse motion com- 
pensation prediction picture on the basis of a current picture corresponding to the input picture signal; means for 
detecting a degree of motion compensation prediction in the second local decoder a selector for selectively outputting 
me first and secoh with the degree of motion compensation prediction; a first 

calculator for calculating a difference between the transform coefficients of; the orthogonal transform circuit and ones 

30 of the first and second transform coefficients which are selected by the selector, and outputting a motion compensation 
prediction error signal; a first quantizer for quantizing the motion compensation prediction error signal from the first 
adder and outputting a first quantized motion compensation prediction error signal; a second calcuteitor for calculating 
a difference between trie second transform coe^^ second local cte^ coefficients 

from the orthogonal transform circuit, and outjputting a second motion compensation prediction error signal; a second 

35 quantizer for quantizing the motion compensation prediction error signal from the second calculator and outputting a 
second quantized motion compensation prediction error signal; and an encoder for encoding the first and second 
quantized motion compensation prediction error signals and outputting encoded signals. 

According to the present invention, there is provided a video encoding apparatus comprising: an orthogonal trans- 
form circuit for dividing an input video signal into a plurality of blocks each containing NxN pixels and orthogonally 

40 transforming the input video signal in units of blocks to obtain a plurality of transform coefficients divided in spacial 
frequency bands; a first motion prediction processing section for performing motion compensation prediction process- 
ing for the plurality of transform coefficients in prdejr to obtain an upper-layer motion compensation prediction signal 
having the number of data enough to obtain a ni^ image quality; a second motion prediction processing section for 
performing motion compensation prediction processing for the plurality of transform coefficients in order to obtain a 

45 lower-layer motion compensation prediction signal upon reducing the number of date; a decision section tor deciding 
in motion compensation on the basis of the lower-layer motion compensation prediction signal whether motion com- 
pensation prediction is correct; a selector for selecting the upper-layer motion compensation prediction signal in re- 
sponse to a decision representing a correct motion compensation prediction from the decision section, and the tower- 
layer motion compensation prediction signal in response to a decision representing an incorrect motion compensation 

so prediction; and an encoder for encoding one of the upper-layer motion compensation prediction signal and the lower- 
layer motion compensation prediction signal which is selected by the selector. 

According to the present invention, there is provided a video encoding apparatus for realizing SNR scalability in 
M layers, comprising: an orthogonal transform circuit for dividing an input video signal into a plurality of blocks each 
containing NXN pixels and orthogonally transforming the input video signal in units of blocks to obtain a plurality of 

ss transform coefficients divided in spactai frequency bands; a first motion compensation prediction processing section 
for performing motion compensation prediction processing for the plurality of transform coefficients in order to obtain 
an mth-layer (m = 2 to M) motion compensation prediction signal; a second motion compensation prediction processing 
section for performing motion compensation prediction processing for the plurality of transform coefficients in order to 
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obtain an (m-1 )th-layer motion compensation prediction signal; switching means for selecting the mth-layer motion 
compensation prediction signal of the first motion compensation prediction processing section in order to obtain an 
mth-layer prediction value when a quantized output from the second motion compensation prediction processing sec- 
tion is 0, and switching between the mth-layer motion compensation prediction signal and the (m-l)th-layer motion 
5 compensation prediction signal in units of transform coefficients in order to select the (m-1 )th-layer motion compensa- 
tion prediction signal when the quantized output is not less than 1. means for calculating a difference signal between 
an (m-1 )th-layer dequantized output from the second motion compensation prediction processing section and an mth- 
layer motion compensation prediction error signal obtained by a difference between the mth-layer motion compensation 
prediction signal and the transform coefficient from the orthogonal transform circuit; and encoding means for quantizing 
io and encoding the difference signal to output an encoded bit stream. 

According to the present invention, there is provided a video encoding/decoding system comprising: a video en- 
coding apparatus for realizing SNR (Signal to Noise Ratio) scalability in M layers, which includes an orthogonal trans- 
form circuit for dividing an input video signal into a plurality of blocks each containing N x N pixels and orthogonally 
transforming the input video signal in units of blocks to obtain a plurality of transform coefficients divided in spacial 
is frequency bands, a first motion compensation prediction processing section for performing motion compensation pre- 
diction processing for the plurality of transform coefficients in order to obtain an mth-layer (m = 2 to M) motion com- 
pensation prediction signal, a second motion compensation prediction processing section for performing motion com- 
pensation prediction processing for the plurality of transform coefficients in order to obtain an (m-1 )th-layer motion 
compensation prediction signal, switching means for selecting the mth-layer motion compensation prediction signal of 
20 the first motion compensation prediction processing section in order to obtain an mth-layer prediction value when a 
quantized output from the second motion compensation prediction processing section is 0, and switching between the 
mth-layer motion compensation prediction signal and the (m-t)th-layer motion compensation prediction signal in units 
of transform coefficients in order to select the (m-1 )th-layer motion compensation prediction signal when the quantized 
output is not less than 1 , means for calculating a difference signal between an (m-1 )th-layer dequantized output from 
2S the second motion compensation prediction processing section and an mth-layer motion compensation prediction error 
signal obtained by a difference between the mth-layer motion compensation prediction signal and the transform coef- 
ficient from the orthogonal transform circuit, and encoding means for quantizing and encoding the difference signal, to 
output an encoded bit stream; and a video decoding apparatus which includes means for extracting codes up to a code 
in the mth (m = 2 to M) layer from the encoded bit stream from the video encoding apparatus, decoding means for 
so decoding the codes of respective layers up to the mth layer, dequantization means for dequantizing, in the respective 
f layers, the quantized values decoded by the decoding means, switching means for switching the mth-layer (m = 2 jo 
M) motion compensation pr ediction value and the (m-1 )th.layer motion connpensati^ in units of trans- 

form coefficients, and outputting the mth-layer motion compensation prediction value for the quantized output of 0 in 
the (m-1 )th layer and the (n>1 )th-layer motion compensation prediction value for the quantized output of not less than 
35 i in the (m-1 jth layer in units of transform coefficients in order to obtain the mth-layer prediction value, and means for 
adding the mth-layer motion compensation prediction value and the (m-1 )th-layer motion compensation prediction 
value to reconstruct the mth-layer motion compensation prediction error signal. 

According to the present invention, there is provided a video encoding apparatus comprising: an orthogonal trans- 
form circuit for dividing an input video signal into a plurality of blocks each containing N x N pixels and orthogonally 
40 transforming an arbitrary-shape picture in units of blocks to obtain a plurality of transform coefficients; means for en- 
coding and outputting an alpha-map signal for discriminating a background of a picture from an object thereof; means 
for calculating an average value of pixel values of an object portion using the alpha-map signal in units of blocks; means 
for assigning the average value to abackground portion of the block; means for deciding using the alpha-map signal 
whether a pixel in the object is close to the background; means for compressing, about the average value, the pixel in 
45 the object decided to be close to the background; and means for orthogonally transforming each block to output an 
orthogonal transform coefficient. 

This invention can be more fully understood from the following detailed description when taken in conjunction with 
the accompanying drawings, in which: 

so FIG. 1 is a block diagram for explaining the present invention, showing the ccwifiguration of an encoding apparatus 

according to the first embodiment of the present invention; 

FIG. 2 is a view for explaining the present invention, which explains a prediction value switching method to be 
applied to the present invention; 

FIGS. 3A and 3B are block diagrams for explaining the present invention, showing the configurations of motion 
S5 compensation prediction sections according to the first embodiment of the present invention; 

FIG. 4 is a block diagram for explaining the present invention, showing the configuration of a decoding apparatus 
according to the first embodiment of the present invention: 

FIG. 5 is a block diagram for explaining the present invention, showing the configuration of an encoding apparatus 
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according. to the second embodiment of the present invention; 

FIGS 6 A and 6B are block diagrams for explaining the present invention, showing the configurations of motion 

compensation prediction sections according to the second embodiment of the present invention; 

FIG. 7 is a^fetock diagram for explaining the present invention, showing the configuration of a decoding apparatus 
5 according to the second embodiment of the present invention. 

FIGS. 8A and 9B are block diagrams for explaining the present invention, showing the configurations of motion 

compensation prediction sections according to the third embodiment of the present invention; 

FIG 9 is a view for explaining the present invention, which illustrates an example of a quantization matnx used in 

the, present invention; - 
10 Flf3 10 is a view for explaining the present invention, which illustrates an example of a quantization matrix used 

in the present invention; 

FIG. 11 •shows an example of a quantization matrix obtained for the example shown ri FIG. 2; 
FIG . 1 2 shows an example of a two-dimensional matrix which is divided into eight portions in each of a horizontal 
direction (h) and a vertical direction (v); 
is FIGVi3 srKWs a scan order for the example shown in FIG 2; 

FIG. 14 is a view tor explaining the present invention, which explains the fourth embodiment of the present inven- 
tion; 

FIGS 15A, 1 5B, and 1 5C are views ifor explaining an example of a video transmission system to which the video 
encoding apparatus„and the video decoding apparatus according to the present invention are applied; 
20 FIG. i 6 is a view for explaining a modification of the second embodiment of the present invention, which is a graph 

showing an example in which an average value is arranged in a background; 

FIG 17 is a view for explaining another modification of the second embodiment of the present invention, which is 
a graph for explaining an example in which a step is decreased: 

FIG 18 is a view for explaining still another modification of the second embodiment of the present invention, which 
25 illustrates examples of block pixel values; 

FIG 1 9 is a view for expiaining still another modification of the second embodiment of the present invention, which 
is a graph for expiaining another example in which a step is decreased; 

FIG . 20 is a view for expiaining still another modification of the second embodiment of the present invention, which 
iirustratese 

30 FIG. 21 is a block diagram showing an example of ah encoding apparatus as still another modification of the second 

embodiment of the present invention; and 

FIG" i^2 is a block diagram showing an example of a decoding apparatus as still another modffica\K>n of the second 
embodiment of present invention; J " ; " " 

35 in the present inVentbn. when motion compensation is to be performed in a transform coefficient domain in units 

of N x N transform coefficients, encodhg in an upper layer (enhancement layer) ts performed on the basis of an already 
decoded and quantized value of a lower layer (base layer) This realizes an encoding system which can perform en- 
coding with little decrease in the encoding coefficient. 

Also, in the above encoding apparatus of the present invention, orthogonal transform can be performed for a picture 
40 domain with an arbitrary shape in accordance with an aipha-map signal indicating the arbitrary-shape picture domain. 
Consequently, a reconstructed picture with a variable image quality can be obtained for an arbitrary-shape picture. 

In the present invention, a frame memory is prepared for each of a background and one or more objects, and 
motion compensation prediction is performed for each of the background and the objects. This improves the efficiency 
of prediction for a portion hidden by overlapping of the objects. 
45 Furth^rrrtore; the%mciency of motion compensation predictive encoding is improved by decreasing the range of 

motion vector detection in the boundary of an object. 

Embodiments of the present invention will be described below with reference to the accompanying drawings. 
The first embodiment of the present invention will be described with reference to FIGS. 1, 2, 3A, 3B : and 4. This 
embodiment is related to an encoding apparatus and a decoding apparatus which realize SNR scalability of M layers 
so as a whole The coding efficiency in the mth layer is improved by adaptrvely switching a motion compensation prediction 
signal in the mth layer and a motion compensation prediction signal in the (m-1)th layer In the accompanying drawings, 
a base layer corresponds to the (m-1 )th layer and an enhancement layer conesponds to the mth layer 

In the encoding apparatus shown in FIG. 1 , an input signal is input to an orthogonal transform circuit, e.g., OCT 
circuit 100: Th© output terminal of the DCT circuit is connected to the input terminals of adders 110 and 111. The other 
55 input termir^i of the adder 110 is connected to a selector 300. The output terminal of the adder 110 ts connected to a 
quantizer 1 30 via ah adder 120. The output terminal of me quantizer 140 is connected to an output buffer 160 via a 
variable-length i encoder 140 and a multiplexer 150 

The output terminal of the quantizer 130 is connected to a motion compensation prediction section (MCP) 200 via 
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a dequantizer 1 70 and adders 1 30 and 1 90. The output of the motion compensation prediction section 200 is connected 
selectively to the adders 110 and 120 by the selector 300 The encoding controller 400 controls the quantizer 1 30 and 
the variable-length encoder 140 in accordance with the output signal from the output buffer 160. 

The output terminal of the adder 111 is connected to the input terminal of the quantizer 131 the output terminal of 

5 which is connected to is connected to the output buffer 161 via a variable-length encoder 141 and a multiplexer 1 51 
The output terminal of the quantizer 131 is connected to a motion compensation prediction section 201 of the 
enhancement layer via a dequantizer 171 and an adder 191 The output terminal of the motion compensation prediction 
section 201 is connected to the selector 300 and adders 1 1 1 and 191. The encoding controller 410 controls the quantizer 
131 and the variable -length encoder 141 in accordance with the output signal from the output buffer 161. A motion 

10 vector detector 500 receives the input video signal 1 0 and is connected to the motion compensation prediction section 
200, motion compensation prediction section 201 and variable-length encoder 141. 

The DCT circuit 100 performs orthogonal transform (DCT) for an input picture signal 1 0 to obtain transform coef- 
ficients of individual frequency components. The adder 110 calculates the difference between the transform coefficient 
from the DCT circuit 100 and one of an output (EMC) from the enhancement layer motion compensation prediction 

is section 200 and an output (BMC) from the base layer motion compensation prediction section 201 which are selectively 
supplied via the selector 300. The adder 120 calculates the difference between an output from the adder 110 and an 
output from the dequantizer 171. 

The quantizer 1 30 quantizes an output from the adder 1 20 in accordance with a quantization scale supplied from 
the encoding controller 400. The variable-length encoder 140 performs variable-length encoding for the quantized 

20 output from the quantizer 1 30 and side information such as the quantization scale supplied from the encoding controller 
400. 

The multiplexer 150 multiplexes the variable-length code of the quantized output and the variable-length code of 
the side information supplied from the variable-length encoder 140. The output buffer 1 60 temporarily holds and outputs 
the data stream multiplexed by the multiplexer 150. 
2S The encoding controller 400 outputs information of an optimum quantization scale Q_scale on the basis of buffer 

capacity information from the buffer 1 60. The encoding controller 400 also supplies this information of the quantization 
scale Q_scale to the variable-length encoder 1 40 as the side information, thereby causing the quantizer 1 30 to perform 
quantization and the variable-length encoder 140 to perform variable-length encoding. 

The dequantizer 170 dequantizes the quantized output from the quantizer 130 and outputs the result The adder 
20 1 80 adds the output from the dequantizer 170 and the output from the dequantizer 171 The adder 190 adds the output 
from the adder t 80 and a compensation prediction value selectively output from the selector 300. 

The motion compensation prediction section 200 calculates a motion compensation prediction value in the en- 
hancement layer on the basis of the output from the adder 180 and a motion vector detected by the motion vector 
detector 500. When receiving the motion compensation prediction value calculated by the motion compensation pre- 
35 diction section 200 and the motion compensation prediction value calculated by the motion compensation prediction 
section 201 , the selector 300 selectively outputs one of these motion compensation prediction values in accordance 
with an output from a binarizing circuit 310. 

In the above configuration, the adder 1 1 0, the adder 1 20, the quantizer 1 30, the variable-length encoder 1 40, the 
multiplexer 150, the output buffer 160, the dequantizer 170, the adder 180, the adder 190, the motion compensation 
40 prediction section 200, the selector 300, and the encoding controller 400 constitute the enhancement layer The quan- 
tizer 170, the adder 180, the adder 190 and the motion compensation prediction section 200 construct a local decoder 
of the enhancement layer. 

The motion vector detector 500 described above receives the same picture signal as the input picture signal to 
the DCT circuit 100 and detects a motion vector from this picture signal. On the basis of the motion vector supplied 
45 from the motion vector detector 500 and the sum output from the adder 191, the motion compensation prediction 
section 201 performs motion compensation prediction and obtains a motion compensation prediction value (BMC) 
which is converted into a DCT coefficient. 

The adder 111 calculates the difference between the output transform coefficient from the DCT circuit 1 00 and the 
output motion compensation prediction value (BMC) from the motion compensation prediction section 201 . The quan- 
so tizer 1 31 quantizes the output from the adder 1 1 1 in accordance with the quantization scale designated by the encoding 
controller 410. 

The binarizing circuit 310 checks whether the quantized value output from the quantizer 131 is "0\ If the value is 
"0", the binarizing circuit 310 outputs '0V If the value is not "0\ the binarizing circuit 310 outputs "1". The dequantizer 
1 71 performs dequantization in accordance with the quantization scale designated by the encoding controller 41 0 .The 
ss adder 1 91 adds the output from the dequantizer 171 and the output from the motion compensation prediction section 
201 and supplies the sum to the motion compensation prediction section 201. 

The variable-length encoder 141 performs variable-length encoding for the quantized output from the quantizer 
1 31 and the side information such as the quantization scale supplied from the encoding controller 410. The multiplexer 
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151 -multiplexes the variable-length code of the quantized output and the variable-length code of the side information 
supplied from the variable-length encoder 141 The output buffer 161 temporarily holds and outputs the data stream 
multiplexed by the multiplexer 151. 

The encoding controller 410 outputs the information of the optimum quantization scale Q_scale on the basis of 
5 buffer capacity information from the buffer 161 . The encoding controller 410 also supplies this information of the quan- 
tization scale Pascal e to the variable-length encoder 141 as the side information,, thereby causing the quantizer 131 
to perform quantization and the variable-length encoder 141 to perform variable-length encoding. 

The adder 111 , the quantizer 131 , the variable-length encoder 141 , the multiplexer 151 , the output buffer 161 , the 
dequantizer 171 ; the adder 191, the motion compensation prediction section 201, the binarizing circuit 310, the en- 
io coding controller 410, and the motion vector detector 500 constitute the base layer The dequantizer 171, the adder 
1 91 and the motion compensation prediction section 201 constitute a local decoder. 

This apparatus with the above configuration operates as follows. 

The input picture signal 1 0 is supplied to the DCT circuit 1 00 and the motion vector detector 500: The motion vector 
detector 500 detects a motion vector from the picture signal 10 and supplies the detected vector to the motion corn- 
is pensation prediction sections 200 and 201 and the variable-length encoder 1.41 . 

the picture signal 10 input to the DCT circuit 100 is divided into blocks each having a size of N x N pixels and 
orthogonally transformed in units of N x N pixels by this DCT circuit 100. Consequently, NXN transform coefficients 
are obtained for each block. These transform coefficients are N X N transform coefficients obtained by separating the 
spacial frequency components of the picture into components ranging from a DC component to individual AC compo- 
20 nents 

These N x N transform coefficients obtained by the DCT circuit 1 00 are supplied to the adder 11 0 in the enhance- 
ment layer and the adder 111 in the base layer. 

In the base layer, the adder 111 calculates the difference between the transform coefficient and the motion com- 
pensation prediction value (BMC) which is converted into a DCT coefficient and supplied from the motion compensation 
25 prediction section 201 , thereby obtaining a prediction error signal. This prediction error signal is supplied to the quantizer 
131 to be quantized in accordance with the quantization scale Q„scale input by the encoding controller 410. The 
quantized prediction error signal is supplied to the variable-length encoder 141 and dequantizer 171 . 

TOe variai>Je-lengtK the 
side ^formation such as the quantization size supplied from the encoding controller 410, and the motion vector infor- 
30 matton supplied from the motion vector detector 500. This vaiiable-lengm-encoded output is supplied to the multiplexer 
1 51 to be multiplexed thereby, and supplied to the output buffer 161. The output buffer 161 outputs the multiplexed 
signal, as an encoded bit stream 21, to a transmission line or a storage medium. Also, the output buffer 161 feedsthe 
capacity oif the buffer back to the encoding c^trqller 410. r r ; : ; : 

In accofdance with the capacity information from the buffer, the encoding controller 410 controls the output from 
35 the quantizer 131 and outputs the quantization scale Q_scale to the quantizer 1 31. This information of the Quantization 
scaie:Q_scale is also supplied to the variable^tength encoder 141 as the side information 

Since the encoding controller 410 controls the output from the quantizer 131 in accordance with the capacity 
information from the buffer, the encoding controller 410 can advance the quantization while controlling the quantization 
scale so that the output buffer 161 does not overflow, 
40 The information of the quantization scale Q_scale is vartable-iength-encoded as the side information by the vari- 

able-length encoder 141 and multiplexed by the multiplexer 1 51 : The multiplexed signal is used as the output from the 
video encoding apparatus. Consequently; the quantization scale used in dequantization when the video decoding ap- 
paratus performs decoding can be obtained. 

Meanwhile; the quantized value of the prediction error signal supplied to the dequantizer 1 71 is dequantized and 
45 supplied to the adder 191 . The adder 191 adds the dequantized value to the motion compensation prediction value 
BMC and thereby calculates a reconstructed value in the transform coefficient domain. This value is supplied to the 
motion compensation prediction section 201 . 

In the enhancement layer, the output EMC from the motion compensation prediction section 200 of the enhance- 
ment layer and the output BMC from the motion compensation prediction section 201 of the base layer are adaptivety 
so and selectively output for each transform coefficient. That is. on the basis of an output BQ from the quantizer 131 in 
the base layer, the selector 300 adaptivery and selectively outputs the output (EMC) from the motion compensation 
prediction section 200 of the enhancement layer and the output BMC from the motion compensation prediction section 
201 of the base layer for each transform coefficient in accoiriarice with a met^ 

The adder 110 calculates a prediction error signal between the transform coefficient of the input picture suppled 
ss from the DCT circuit 100 and an output EP from the selector 300 and supplies trie signal to the adder 120. The adder 
1 20 calculates the difference between a signal 30 of the dequantized value BQ supplied from the dequantizer 171 and 
the output from the adder 1 10 and supplies the difference as a difference value output signal EC to the quantizer 130 
This difference value output signal EC is the motion compensation prediction error signal. 
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The quantizer 130 quantizes the difference value output signal EC in accordance with the quantization scale 
Q_scaie supplied from the encoding controller 400 and supplies the quantized signal to the variable-length encoder 
1 40 and the dequantizer 1 70. 

The. variable-length encoder 140 performs variable-length encoding for the quantized motion compensation pre- 
s diction error signal together with the side information and supplies the encoded signals to the multiplexer 150. The 
multiplexer 150 multiplexes these signals and supplies the multiplexed signal to the output buffer 160. 

The output buffer 160 outputs the multiplexed signal to a transmission line or a storage medium as an encoded 
bit stream 20 for the enhancement layer. Also, the output buffer 1 60 feeds the capacity of the buffer back to the encoding 
controller 400. 

10 The quantized value supplied to the dequantizer 170 is dequantized. The adder 180 adds the dequantized value 

to the output 30 supplied from the dequantizer 1 71 of the base layer, thereby reconstructing the prediction error signal. 

The adder 190 adds the prediction error signal reconstructed by the adder 180 to the motion compensation pre- 
diction value EMC and thereby calculates a reconstructed value in the transform coefficient domain. This reconstructed 
value is supplied to the motion compensation prediction section 200. 

is FIG. 2 shows a switching unit described in reference (T K. Tan et al., "A Frequency Scalable Coding Scheme 

Employing Pyramid and Subband Techniques", IEEE Trans CAS for Video Technology, Vol. 4, No. 2, Apr. 1994), which 
is an example of the switching unit optimally applicable to the selector 300 

Referring to FIG. 2, the binarizing circuit 310 decides whether the value of the output BQ from the quantizer 131 
in the base layer is "p.". This decision result is supplied to the selector 300. If the value of the output BQ from the 

20 quantizer 1 31 is "0*, the selector 300 selects the transform coefficient output EMC from the enhancement layer motion 
compensation prediction section 200. If the value is "1 the selector 300 selects the transform coefficient output BMC 
from the base layer motion compensation prediction section 201. 

That is, the binarizing circuit 31 0 outputs "0" when the value of the output BQ from the quantizer 1 31 in the base 
layer is •O - and outputs *1" when the value is hot "0*. Therefore, the selector 300 is made to select EMC when the 

2S output from the binarizing circuit 310 is *0" and BMC when the output is *1'. Consequently, the transform coefficient 
output EMC from the motion compensation prediction section 200 in the enhancement layer is applied to a transform 
coefficient in a position where the output BQ from the quantizer 1 31 is m 0\ and the transform coefficient output BMC 
from the motion compensation prediction section 201 in the base layer is applied to a transform coefficient in a position 
where the output BQ from the quantizer 1 31 is not "0 B . 

30 The quantizer 131 in the base layer receives the output from the adder 111 and quantizes this output from trje 

; adder rl 1 1 , The aoVder 1 : 1 1 receives the output from the DCT circuit 1 00 and the motion compensation prediction val&e 
obtained by the motion condensation prediction section 201 f rom a picture^ in an imrnediately previous frame, and 
calculates the difference between them. Therefore, if the calculated rnotion compensation prediction value is correct, 
the difference between the two values output from the adder 111 is "0". 

3S Accordingly, of the quantized values as the output BQ from the quantizer 1 31 in the base layer, coefficients (values 

in portions enclosed by the circles in FIG. 2) having values other than are coefficients whose motion compensation 
prediction is incorrect. 

If the motion compensation prediction section 200 performs motion compensation prediction by using the same 
motion vector as in the base layer supplied from the motion vector detector 500, it is estimated that motion compensation 
40 prediction for coefficients (values. in portions enclosed by the circles) in the enhancement layer in the same positions 
as in the base layer is incorrect. 

Accordingly, the selector 300 selects BMC for these coefficients. 

On the other hand, it is estimated that motion compensation for other coefficients is correct. Therefore, the selector 
300 selects a prediction value in the enhancement layer with a smaller encoding deviation. Consequently, the signal 
45 EC encoded in the enhancement layer is used as the quantized error signal of the base layer when motion compensation 
prediction is incorrect, and as the motion compensation prediction error signal of the enhancement layer when motion 
compensation prediction is correct This improves the coding efficiency of coefficients whose motion compensation 
prediction is incorrect. 

Note that the technique disclosed in the reference cited above is based on the assumption that pictures having 
so low resolutions are reconstructed in the base layer, and so low-frequency coefficients which are 1/4 the transform 
coefficients calculated by the DCT circuit 100 are separated and supplied to the base layer As a consequence, the 
reliability of estimation for switching prediction for each transform coefficient is decreased due to an error produced by 
resolution conversion. 

In this embodiment, on the other hand : the resolutions of the base layer and the enhancement layer are equal. 
55 Therefore, the embodiment is different from the technique disclosed in the reference cited above in that the accuracy 
of estimation is improved. A great advantage of the embodiment is a high image quality. 

The configuration of the motion compensation prediction sections 200 and 201 used in the apparatus of the present 
invention will be described below. 
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FIG. 3 A is a block diagram showing the configuration of the motion compensation prediction sections 200 and 201 
Each of the motion compensation prediction sections 200 and 201 consists of an IDCT circuit 210. a frame memory 
220. a motion compensation circuit 230, and a OCT circuit 240. 

The IDCT circuit 210 restores the reconstructed picture signal by performing inverse orthogonal transform (IDCT) 
s for the output from the adder 1 90 or 1 91. The frame memory 220 holds the reconstructed picture signal obtained by 
this inverse orthogonal transform, as a reference picture, in units of frames. The motion compensation circuit 230 
extracts a picture in a position indicated by a motion vector in units of blocks from the picture signals (reference pictures) 
stored in the frame memory 220 . The DOT circuit 240 performs orthogonal transform {DCT) for the extracted picture 
and outputs the result. Note that the motion vector is supplied from the motion vector detector 500. In this configuration, 
io a reconstructed value in a transform coefficient domain is inversely transformed into the reconstructed picture signal 
by the IDCT circuit 210 and stored in the frame memory 220. The motion compensation circuit 230 extracts a picture 
in a position indicated by the motion vector in units of blocks from the reference pictures stored in the frame memory 
220, and supplies the extracted picture to the DCT circuit 240. The DCT circuit 240 performs DCT tor the supplied 
picture and outputs the result as a motion compensation prediction value in the DCT coefficient domain. 
is in this manner, the motion compensation prediction value in the DCT coefficient domain can be obtained. 

The foregoing is the explanation of the encoding apparatus. The decoding apparatus will be described below 
FIG 4 is a block diagram of the decoding apparatus according to the first embodiment of the present invention. 
According to the present decoding apparatus, a buffer 162 on the enhancement layer side receives a coded bit 
stream sent from the encoding apparatus. The output terminal of the buffer 162 is connected to variable-length encoder 
20 1 42 via a demultiplexer 1 52 the output terminal of which is connected to a dequantizer 1 72. The output terminal of the 
dequantizer 1 72 is connected to a motion compensation prediction section 202 via adders 181 and 192. The output 
terminal of the motion compensation prediction section 202 is connected an adder 192 via a selector 300. 

The buffer 163 on the base layer side receives the encoded bit stream 23 sent from the encoding apparatus. The 
output terminal of the buffer 163 is connected to a variable-length decoder 143 via a segmentation circuit 153. The 
2S output terminal pf the variable length decoder 143 is connected to a motion compensation prediction section 203 via 
a dequantizer .173 and an adder 193 and to the switch control terminal of a selector 300 via a binarizing circuit 310. 
The output terminal of the motion compensation; prediction section 203 is connected to an adder 1 93 and the selector 

300; , 'V-v- .' 

: TT^e input; buff er 1 62V the demultiplexer 1 52, the variable-length decoder 1 42, #ie dequantizer 1 72, the adders 181 

30 and 1 92; the selector 300. and the motion compensation prediction section 202 constitute an enhancement layer The 
input buffer 1 63. the demultiplexer 1 53, the variable-length decoder 143, the dequantizer 173, the adder 193, the 
binarizing circuit 310, and the motion compensation prediction section 203 constitute a base layer. 

The input buffer 1 62 in the enhancement layer receives and temporary holds an emxxted multiplexed bit stream 
22 in the enhancement layer The demultiplexer 152 demultiplexes the bit stream 22 obtained via the input buffer 162, 

35 i.e.; demultiplexes the multiplexed signal into trie ortg^l sigrals, thereby restoring encoded ^formation of side for- 
mation arid encoded information of a difference value output signal EC of a picture. 

The variable-length decoder 1 42 performs variable-length decoding for the encoded signals demultiplexed by the 
demultiplexer 152 to thereby restore the original side information and the difference value output signal EC of the 
picture. On the basis of the information of a quantization scale Q_scale of the restored side information, the dequantizer 

40 172 dequantizes the c^erence value output signal EC of the picture from the variable-length decoder 142 and outputs 
the dequantized signal. The adder 181 adds the dequantized signal and the dequantized output from the dequantizer 
1 73 for the base layer. 

The adder 192 adds the output from the adder til and the output EP from the selector 300 and outputs the sum. 
The motion con^rtsation prediction section 202 receives the output from the adder 192 and the decoded difference 
45 value output signal EC of the picture which is the output from the variable-length decoder 1 43 for the base layer and 
obtains a motion compensation prediction value EMC. The output motion compensation prediction value EMC from 
the motion compensation prediction section 202 is used as an enhancement layer output 40 and as one input to the 
selector 300. \ 

The selector 300 receives the output (motion compensation prediction value EMC) from the motion compensation 
so prediction section 202 for the enhancement layer and the output from the motion ccmnpensation prediction section 203 
for the base layer. In accordance with the output from toe binarizing circuit 310, the selector 300 selectively outputs 
one of these two inputs. 

The input buffer 163 receives and temporarily holds an encoded and multiplexed bit stream 23 for the base layer 
The demultiplexer 153 demultiplexes the bit stream 23 obtained via the input buffer 163, Le., demultiplexes the multi- 
ss plexed signal into the original signals, thereby restoring encoded information of the side information and encoded 
information of the difference value output signal EC of the picture. 

The variable-length decoder 1 43 performs variable-length decoding for the encoded signals demultiplexed by the 
demultiplexer 153 to thereby restore the original side information and the difference value output signal EC of the 
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picture. On the basis of the information of the quantization scale Q_scale of the restored side information, the dequan- 
tizer 173 dequantizes the difference value output signal EC of the picture from the variable-length decoder 143 and 
supplies the dequantized signal to the adders 1 8 1 and 1 93. The adder 1 93 adds the dequantized signal and the motion 
compensation prediction value EMC supplied from the motion compensation prediction section 203 for the base layer. 

The motion compensation prediction section 203 receives the output from the adder 193 and the motion compen- 
sation prediction value EMC, which is the output of an immediately previous frame from the section 203, and obtains 
the motion compensation prediction value EMC of the current frame. The output motion compensation prediction value 
EMC from the motion compensation prediction section 203 is used as an output 41 of the base layer and as the other 
input to the selector 300. 

The operation of the decoding apparatus with the above configuration will be described below. In this apparatus, 
the base layer bit stream 23 is supplied to the input buffer 163 and the enhancement layer bit stream 22 is supplied to 
the input buffer 1 62. 

The input base layer bit stream 23 is stored in the input buffer 163 and supplied to the demultiplexer 153. The 
demultiplexer 1 53 demultiplexes the signal in accordance with the type of the signal. That is, the bit stream 23 is formed 
by multiplexing signals of the side information such as the quantized value of a transform coefficient, the motion vector, 
and the quantization scale. Upon receiving the bit stream 23, therefore, the demultiplexer 153 demultiplexes the bit 
stream into the original codes such as the quantized value of the transform coefficient, the motion vector, and the 
quantization scale Q_scale in the side information. 

The codes demultiplexed by the demultiplexer 153 are supplied to the variable-length decoder 143 and decoded 
into signals of the quantized value of the transform coefficient, the motion vector and the quantization scale Q_scale. 
Of the decoded signals, the motion vector is supplied to the motion compensation prediction section 203, and the 
quantized value of the transform coefficient and the quantization scale Q_scale are supplied to the dequantizer 173. 
The dequantizer 173 dequantizes the quantized value of the transform coefficient in accordance with the quantization 
scale Q_scale and supplies the dequantized transform coefficient to the adder 193. 

The adder 1 93 adds the dequantized transform coefficient and the motion compensation prediction value in the 
transform coefficient domain supplied from the motion compensation prediction section 203, thereby calculating the 
reconstructed value in the transform coefficient domain. 

This reconstructed value is supplied to the motion compensation prediction section 203. The configuration of the 
motion compensation prediction section 203 is as shown in FIG 3B. That is, the reconstructed value supplied from 
the adder 193 is inversely orthogonally transformed by an IDCT circuit 210 in the motion compensation prediction 
section 203 and output as the reconstructed picture signal 41 . The signal is also stored in a frame rnemory 220 in r^e 
motion compensation prediction section 203. : 

In the motion compensation prediction section 203, on the basis of the supplied rnotion vector described above, 
a motion compensation circuit 230 extracts a picture in a position indicated by the motion vector in units of blocks from 
the picture signals (reference pictures) stored in the frame memory 220. A DCT circuit 240 performs orthogonal trans- 
form (DCT) for the extracted picture and outputs the result as a transform coefficient output BMC to the adder 193 and 
the selector 300. 

Meanwhile, the enhancement layer bit stream 22 is supplied to the enhancement layer This bit stream 22 is stored 
in the enhancement layer input buffer 162 and supplied to the demultiplexer 152. 

The demultiplexer 152 demultiplexes the bit stream 22. That is, the bit stream 22 is formed by multiplexing signals 
of the side information such as the quantized value of a transform coefficient, the motion vector, and the quantization 
scale Q_scale. Upon receiving the bit stream 22, therefore, the demultiplexer 1 52 demultiplexes the bit stream into the 
original codes such as the quantized value of the transform coefficient, the motion vector and the quantization scale 
O scale 

" The codes demultiplexed by the demultiplexer 152 are supplied to the variable-length decoder 142 and decoded 
into signals of the quantized value of the transform coefficient, the motion vector, and the like. Of the decoded signals, 
the motion vector is supplied to the motion compensation prediction section 202, and the quantized vaiue of the trans- 
form coefficient and the quantization scale Q_scale are supplied to the dequantizer 1 72. The dequantizer 172 dequan- 
tizes the quantized value of the transform coefficient in correspondence with the quantization scale Q_scale and sup- 
plies the dequantized transform coefficient to the adder 181 . The dequantized value is added to a dequantized value 
31 of the base layer supplied from the dequantizer 1 73, and the sum is supplied to the adder 1 92. 

The adder 193 adds the output from the adder 181 and a signal EP supplied from the selector 300 to thereby 
calculate the reconstructed value in the transform coefficient domain. This reconstructed value is supplied to the motion 
compensation prediction section 202. The configuration of the motion compensation prediction section 202 is as shown 
in FIG. 3B. That is, the reconstructed value supplied from the adder 193 is inversely orthogonally transformed by an 
IDCT circuit 210 in the motion compensation prediction section 202 and output as a reconstructed picture signal 40. 
The signal is also stored in a frame memory 220 in the motion compensation prediction section 202. 

in the motion compensation prediction section 202, on the basis of the supplied motion vector described above a 
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motion compensation circuit 230 extracts a picture in a position indicated by the motion vector in units of blocks from 
the picture signals (reference pictures) stored in the frame memory 220. A DCT circuit 240 performs orthogonal trans- 
form (DCT) for the extracted picture and outputs the result as a transform coefficient output BMC to the adder 1 93 and 
the selector 300, 

5 The. selector 300 receives the decision result from the binarizing circuit 310 and selects one of BMC and EMC. 

That is, the binarizing circuit 310 receives an output BQ from the variable-length decoder 143 and decides whether 
the value is *0" This decision result is supplied to the selector 300. 

If the value of the output BQ from the variable -length decoder 143 is "0", the selector selects the transform coef- 
ficient biitput EMC from the motion compensation prediction section 202. If the value is "1 *, the selector 300 selects 

w the^ transform coefficient output BMC from the motion compensation prediction section 203. 

That is, the binarizing circuit 310 outputs B 0' when the value of the output BQ from the variable-length decoder 
143 in the base layer is "0" and outputs "1" when the value is not "0". Therefore, the selector 300 is made to select 
EMC when the output from the binarizing circuit 310 is "0" and BMC when the output is "1\ Consequently, the transform 
coefficient output EMC from the motion compensation prediction section 202 in the enhancement layer is applied to a 

is transform coefficient in a position where the output BQ from the variable-length decoder 1 43 is "0", and the transform 
cciefticient output BMC from the motion compensation prediction section 203 in the base layer is applied to a transform 
coefficient in a position where the output BQ from the variable-length decoder 1 43 is not JO". 

Thei ^output from the variable^iength decoder 143 in the base layer contains the motion compensation prediction 
error signal and the motion vector obtained.ori the encoding side. When the motion compensation prediction error 

20 signal arid the motion vector are supplied to the motion compensation prediction section 203, the motion compensation 
prediction, section 203 obtains the motion compensation prediction error between the picture of the immediately pre- 
vious frame and the current picture. 

Meanwhile- the binarizing circuit 31 6 receives the decoded base layer motion compensation prediction value signal 
from the variable-length decoder 1 43. If the signal value is -0". the binarizing circuit 310 outputs "0' to the selector 300. 

25 If the signal value is not "0", the binarizing circuit 310 outputs M" to the selector 300. 

If me c^tput from the binarizing circuit 310 is "0% the selector 300 selects the output EMC with a smaller encoding 
deviation from the enhancement layer motion compensation prediction section 203. If the output from the binarizing 
circuit 310 is "1", the selector 300 selects the transform coefficient output BMC with a larger encoding deviation from 
tha base tayer r^ion compensation prediction section 202. 

so : Eventually- if the rXJT co^ base layer motion compensation prediction is "0". the output 

from the ^motion compensatiar) prediction section 202 which is the reconstructed value of the transform coefficient 
cirtpul EMC from the enhancement layer motion cc*npensation prediction section 200. is. selected: If the error is M", 
the output ^rom the motion compensation prediction section 203 which is the reconstructed value of the transform 
coefficient output BMC from the base layer motioh compensation prediction section 201 is selected. 

35 This processing is analogous to the processing in the encoding apparatus. Accordingly: as the transform coefficient 

output of ^ motion compensation prediction in the enhancement layer, as in the selection done on the encoding side, an 
output Jorthe base layer is used in a portion where motion cxxT^nsation prediction is incorrect, and an output for the 
enhancement layer with a smaller encoding deviation is used in a portion where the prediction is correct. Consequently, 
following mis switching the encoding apparatus can smoothly reconstruct pictures. 

40 In the first embodiment described above, each frame of a video picture, is divided into matrix blocks each having 

a predetermined number (N x N) of pixels arid orthogonally transformed to obtain transform coefficients of individual 
spacial frequency bands. For each of the N x N transform coefficients thus obtained, motion compensation is performed 
in the domain of the transform coefficient in upper and lower layers. When motion compensation is to be performed in 
this video encoding, whether motion compensation prediction is correct is checked on the basis of an already decoded 

45 and quantized value in the lower layer (base layer). If the motion compensation prediction is correct, the upper layer 
(enhancement layer) is encoded by using a motion compensation prediction value with a smaller encoding deviation 
obtained for the upper layer. If the motion compensation prediction is incorrect, the upper layer is encoded by using a 
motion cc<npensation prediction value obtained for the lower layer (base layer) and having a larger encoding deviation 
than that for the enhancement layer This improves the coding efficiency of a coefficient the motion compensation 

so prediction for which is incorrect and thereby realizes an encoding system capable of encoding with little decrease in 
the coding efficiency. 

The foregoing is an embodiment in which a whole video picture is efficiently encoded in the scalable encoding 
method. An embodiment in which the present invention is applied to arbitrary-shape picture encoding by which a back- 
ground and an object in a video picture are separately encoded will be described below. This second embodiment of 
ss the present invention will be described with reference to FIGS. 5, 6A, 6B, and 7. In this embodiment, the technique of 
the first embodiment is applied to pictures having arbitrary shapes represented by alpha-map signals. 

FIG. 5 shows an encoding apparatus of the present invention as the second embodiment The basic configuration 
of this encoding apparatus is the same as the encoding apparatus explained in the first embodiment Accordingly, the 
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same reference numerals as tn the configurations shown in FIGS. 1 and 4 denote the same parts and a detailed 
description thereof will be omitted 

This configuration differs from FIG. 1 in eight points: that is. an arbitrary shape orthogonal transform circuit 101 is 
provided instead of the DCT circuit 100. inputs are received via a frame memory 700, an encoding controller 420 is 
provided instead of the encoding controller 400, an encoding controller 430 is provided instead of the encoding controller 
410, a motion compensation prediction section 600 is provided instead of the motion compensation prediction section 
200 for an enhancement layer, a motion compensation prediction section 601 is provided instead of the motion com- 
pensation prediction section 201 , a motion vector detector 510 is provided instead of the motion vector detector 500, 
and a multiplexer 155 is provided instead of the multiplexer 151. 

The frame memory 700 temporarily holds an input picture signal in units of frames. The arbitrary shape orthogonal 
transform circuit 101 extracts an object region from the pictures stored in the frame memory 700 by referring to a 
separately supplied alpha-map. The circuit 101 divides the rectangle region including the object region into blocks of 
a predetermined pixel size and performs DCT for each block. 

The encoding controller 420 refers to the alpha-map and generates a quantization scale Q_scale t which gives an 
enhancement layer optimum quantization scale to output buffer capacity information from an output buffer 160, and 
side information. The encoding circuit 430 refers to the alpha-map and generates a quantization scale Q_scale, which 
gives a base layer optimum quantization scale and side information to output buffer capacity information from an output 
buffer 161; and side information. 

The motion compensation prediction section 600 refers to the alpha-map and performs motion compensation pre- 
diction for a picture in the interest region part on the basis of a reconstructed value in a transform coefficient domain 
supplied from an adder 190 and a reconstructed value in an immediately previous frame. The motion compensation 
prediction section 601 refers to the alpha-map and performs motion compensation prediction for the picture in the 
interest region part on the basis of a reconstructed value in the transform coefficient domain supplied from an adder 
191 and the reconstructed value in the immediately previous frame. 

The motion vector detector 510 refers to the alpha-map and detects a motion vector in the picture in the interest 
region part from the pictures stored in the frame memory 700. 

The multiplexer 155 is provided for the base layer The multiplexer 155 multiplexes a variable-length code o^a 
prediction error signal from a variable-length encoder 141 , a variable-length code of side information such as mode 
inforrnation. containing quantization scale information, a variable-length code of a motion vector, and a code (alpha- 
code) of a separately supplied alpha-map, and supplies the multiplexed signal to the output buffer 161 

In this apparatus with the above configuration, an input picture signal 1 0 is temporarily stored in the frame memory 
700 and read out to the arbitrary shape orthogonal transform circuit 101 and the rrotion vector detector 510. In addition 
to the picture signal 10, an alpha-map signal 50 which is a map information signal for distinguishing a background 
portion from an object portion in a picture is input to the arbitrary shape orthogonal transform circuit 101 

This alpha-map signal can be acquired by applying, e.g., a chromakey technique. For example, in the case of an 
alpha-map for distinguishing a person (object) from a background, the image of the person is taken by the chromakey 
technique and btnarized to obtain a bitmap picture in which the person image region is "1 " and the background region 
is *0V This picture can be used as an alpha-map. 

The arbitrary-shape orthogonal transform circuit 101 refers to this alpha-map signal, checks where the object 
region of the picture is, divides the rectangle region including the object region into square blocks each consisting of 
N x N pixels, and orthogonally transforms each block to obtain NXN transform coefficients. As a technique to or- 
thogonally transform an arbitrary-shape region of a picture by using an alpha-map, it is only necessary to use a tech- 
nique established by the present inventors and disclosed in above-mentioned Japanese Patent Application No. 7-97073 
which is already filed. 

Although the explanation of the operation of the encoding apparatus according to the second embodiment has not 
finished, processing of decreasing a step will be described below as a modification. 

In the method of the second embodiment described above, the average value of the object is arranged in the 
background. In addition to this processing, if pixel values of the object are compressed around the average value by 
a predetermined scaling coefficient, the step of a pixel value in the boundary between the object and the background 
can be decreased. Details of this processing will be described below. 

To decrease the step of a pixel value in the boundary between the object and the background, pixel values of the 
object are compressed around the average value by a predetermined scaling coefficient. Examples of the method are 
illustrated in FIGS. 1 6 and 1 7. Athough actual pictures are two-dimensional signals, one-dimensional signals are shown 
for simplicity. In these drawings, a pixel value is plotted on the ordinate, and a pixel row is plotted on the abscissa. The 
left-hand side of the position of a pixel row e is an object region, and the right-hand side is a background region. FIG. 
16 shows a state in which a pixel value average value a of the object is arranged in the background portion. 

FIG. 17 shows the result of compression around the pixel value average value a. Assuming the luminance before 
the compression is x and the luminance after the compression is v., the luminance % after the compression can be 
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represented by where c is a constant between "0" and "1 ". 

y = c x (x - a) + a 

5 " 

Compression can be perlormed for all pixels x I to x23 in an object shown in FIG. 18. However, the step can also 
be decreased by compressing only the pixels xl to x8 in the object close to the boundary to the background portion 
Although an additional arithmetic operation is necessary to check whether a pixel is close to the boundary, the method 
has the advantage that the pixels x9 to x23 in the object not in contact with the boundary to the background are kept 
10 unchanged. 

In this modification, it is decided that, of pixels in the object those in contact with the background in any of the 
upper, lower, left, and right portions are pixels close to the boundary. 

The foregoing is the modification by which pixel values in the object are compressed around the average value by 
a. predetermined, scaling coefficient in order to decrease the step of a pixel value in the boundary between the object 
is and the background However, the step of a pixel value in the boundary between the object and the background can 
also be decreased by processing the background portion. This modification will be described below. 

FIG. 19 shows the modification of processing the background portion. In this modificat ion, of pixels in the back- 
ground, the values of pixels close to the object region are so altered as to decrease the step. A practical example is 
shown in FIG 20. Referring to FIG. 20, xri (n = i to 23) indicates a pixel value in the object andan <n = 1 to 41 ) indicates 
20 a pixel value in the background. Before the processing, all pixel values an in the background are equal to a pixel 
average value a. 

First, the values of background pixels a1 to a9 in contact with pixels in the object region in any of the upper/ lower, 
left, and right portions are replaced with average values of the values of the contacting object pixels xn and their own 
pixel values For example, the background pixel al is replaced with "(a 1 + x1 )/2\ and the background pixel a3 is replaced 
2$ with "(a3 + x2 + x3>/3". 

Subsequently, the background pixels a10 to a17 in contact with the background pixels a1 to a9 are similarly re- 
placed; As an example, a1 0 is replaced with "(ai 0 + a1 )/2". As the background pixel al, the previously replaced value 
is used; 

Likewise, the background pixels a18 to a24 are sequentially replaced As a consequence, the steps of pixel values 

so in the boundary between the object and the background are decreased. 

When pixel values are altered as described above, the average value of the bikx:kcte Therefore, pixel values 
in the background portion can also be corrected so that the original block average value remains unchanged. This 
correction is to; add or subtract a predetermined value to or from all pixels in the background or to alter pixel values in 
the background far frorri the boundary in a direction opposite to the luminance direction in which pixel values close to 

35 /the boundary are altered; 

When pixel values in the object are altered, a, picture close to an input picture can be obtained by restoring the 
portion before the alteration after the picture is decoded. For example, in the above method in which compression is 
performed around the average value, alteration is done as follows assuming that a decoded value of a pixel compressed 
in encoding is yd and a pixel value after the alteration is xd: where ad is the average value of the object or the background 

40 of the decoded picture or the average value of the whole block. Although yd often takes a value somewhat different 
from y due to an encodincydecodinig distortion, xd close to x can be obtained by this alteration. 

According to the encoding apparatus of the above modifications, as shown m FIG. 21 , a picture signal is input to 
me input terminal of a switch 2003 one output terminal of which is connected to a switch 2009, a cornpressor 2010 
and an average Circuit 2011 . The output terminal of the compressor 2010 is connected to the other input terminal of 

45 the switch 2009. The output terminal of the switch 2005 is connected to the switch 2005. The output terminal of the 
average circuit 2011 is connected to the compressor 2010 and the switch 2005. 

A decision circuit 2004 receives an alpha-map signal and the output terminal of the decision circuit 2004 is con- 
nected to the control terminal of the switch 2009. An encoder 2006 receives the alpha-map signal and a DCT circuit 
2017 receives a signal selected by the switch 2005. 

so in the encoding apparatus constructed as described above, the encoder 2006 encodes an externally input alpha- 

map signal 2001. The switch 2003 receives the alpha-map signal 2001 and a picture signal 2002. On the basis of the 
alpha-map signal 2001 , the switch 2003 divides the picture signal 2002 into an object picture 2007 and a background 
picture 2008 and discards the background picture 2008. "Discarding" does hot necessarily mean "sending the picture 
to some other place" but simply means that the picture is left unused after that. 

ss The decision circuit 2004 decides on the basis of the alpha-map signal 2001 whether an interest pixel which is a 

pixel currently being processed in the object picture 2007 supplied via the switch 2003 is in contact with the background. 
The decision circuit 2004 supplies a decision result 201 3 to the switch 2009. 

The average circuit 201 1 calculates an average vaiue.201 2 of the object picture 2007 supplied via the switch 2003 
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and outputs the average value 2012 to the compressor 2010 and the switch 2005. The compressor 2010 compresses 
the amplitude of the object picture 2007 around the average value 2012 to obtain a compressed picture 2014 and 
outputs the compressed picture 2014 to the switch 2009. 

The switch 2009 receives the compressed picture 2014 and the object picture 2007 from the switch 2003 and 
5 refers to the decision result 201 3 from the decision circuit 2004. If the interest pixel is a pixel in a portion in contact 
with the background, the switch 2009 selectively outputs the compressed picture 2014 as an encoded picture 2015 of 
the object. If the interest pixel is not in contact with the background, the switch 2009 selectively outputs the object 
picture 2007 as the. encoded picture 2015 of the object. 

The switch 2005 receives the alpha-map signal 2001, the encoded picture 2015 of the object supplied from the 
10 switch 2009, and the average value 201 2 calculated by the average circuit 2011 . On the basis of the input alpha-map 
signal 2001, if the interest pixel which is a pixel currently being processed is the object, the switch 2005 selectively 
outputs the encoded picture 201 5 of the object as an encoded picture 2016. If the interest pixel is the background, the 
switch 2005 selectively outputs the average value 201 2 as the encoded picture 2016. 

The DCT circuit 2017 performs DCT for the output encoded picture 2016 from the switch 2005 and outputs a 
*5 transform coefficient 2018. 

In the encoding apparatus with the above configuration, the alpha-map signal 2001 and the picture signal 2002 
are externally input. The alpha-map signal 2011 is supplied to the switch 2003, the decision circuit 2004, the switch 
2005, and the encoder 2006. The picture signal 2002 is supplied to the switch 2003. 

On the basis of the input alpha-map signal 2001, the switch 2003 divides the picture signal 2002 into the object 
20 picture 2007 and the background picture 2008 and discards the background picture 2008. As described previously 
"discarding" does not necessarily mean 'sending the picture to some other place" but simply means that the picture is 
left unused after that. 

The object picture 2007 separated by the switch 2003 is supplied to the switch 2009. the compressor 2010, and 
the average circuit 2011 . The average circuit 2011 calculates the average value 201 2 of the object picture and supplies 
2S the average value 2012 to the compressor 2010 and the switch 2005. The compressor 2010 compresses the amplitude 
of the object picture 2007 around the average value 2012 and supplies the compressed picture 2014 obtained by this 
compression to the switch 2009. 

The decision circuit 2004 which has received the alpha-map signal 2001 decides whether an interest pixel which 
is a pixel currently being processed is in contact with the background, and supplies the decision result 201 3 to the 
30 switch 2009. If the decision result 201 3 from the decision circuit 2004 indicates that the interest pixel is in contact with 
the background, foe switch 2009 selectively outputs the compressed picture 2014 as the encoded picture 2015 of She 
object. If the interest pixef is not in contact with the background, the switch 2009 selectively outputs the object picture 
2007 as the encoded picture 2015 of the object. 

The output encoded picture 2015 of the object from the switch 2009 is supplied to the switch 2005. The switch 
35 2005 refers to the alpha-map signal 2001 and, if the interest pixel is the object, selectively outputs the encoded picture 
2015 of the object as the encoded picture 2016. If the interest pixel is the background, the switch 2009 selectively 
outputs the average value 2012 as the encoded picture 2016. 

The encoded picture 2016 output from the switch 2005 is supplied to the DCT circuit 2017. The DCT circuit 2017 
performs OCT for the encoded picture 201 6 to obtain the transform coefficient 20 1 8 and outputs the transform coefficient 
40 201 8 to the outside. The alpha-map signal 2001 is encoded by the encoder 2006 and output to the outside as an alpha- 
code 201 9. 

Note that there is another method in which an alpha-map is encoded before a picture is encoded and the decoded 
signal is input to the switches 2003 and 2005 and the decision circuit 2004. If a distortion occurs in encoding and 
decoding of an alpha-map, the alpha-map signals on the encoding and decoding sides can be made equal by this 
45 method. 

FIG. 22 shows a decoding apparatus as a counterpart of the encoding apparatus in FIG. 21. According to this 
decoding apparatus, a decoder 2020 receives an encoded alpha-map signal. The output terminal of the decoder 2020 
is connected to control terminals of switches 2023and 2025 and a decision circuit 2024. An inverse DCT circuit 2021 
receives transform efficient 2018 of the encoded picture. The output terminal of the inverse DCT circuit 2021 is con- 
50 nected to a decompressor 2030 and an average circuit 2031 . The output terminal of the decompressor 2030 is con- 
nected to a switch 2029 together with the output terminal of the switch 2023. The switch 2029 is connected to a switch 
2036. 

In the decoding apparatus as described above, the decoder 2020 receives the externally input alpha-code 2019, 
decodes the alpha-code 2019, and supplies a decoded alpha-map signal 2022 to the switch 2023, the decision circuit 
55 2024, and the switch 2025. The inverse DCT circuit 2021 performs inverse DCT for the externally input transform 
coefficient 2018 to decode a picture and supplies the picture obtained by the decoding, i.e., a decoded picture 2026, 
to the switch 2023 

The decision circuit 2024 decides on the basis of the alpha-map signal 2022 decoded by the decoder 2020 whether 
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an interest pixel in an object picture 2027 is in contact with the background. The decision circuit 2024 outputs a decision 
result 2034 to the switch 2029 

On the basis of the alpha-map signal 2022 decoded by the decoder 2020. the switch 2023 divides the decoded 
picture 2026 supplied from the inverse DCT circuit 2021 into the object picture 2027 and a background picture 2028 
5 The switch 2023 outputs the object picture 2027 to the switch 2029, the decompressor 2030, and the average circuit 
2031 and discards the background. picture 2028 

The average circuit 2031 calculates an average value 2032 of the object picture 2027 supplied from the switch 
2023 and outputs the average value 2032 to the decompressor 2030. The decompressor 2030 expands the amplitude 
of the object picture 2027 around the average value 2032 to obtain an expanded picture 2033 and outputs the expanded 
io picture 2033 to the switch 2029. 

Of the object picture 2027 and the expanded picture 2033 thus input, the expanded picture 2033 is selectively 
output as a decoded picture 2035 of the object from the switch 2029 to the switch 2025, if the output decision result 
2034 from the decision circuit 2033 indicates that the interest pixel is in contact with the background. If the interest 
pixel is not in contact with, the background, the switch 2029 selectively outputs the object picture 2027 as the decoded 
is picture 2035 of the object to the switch 2025. 

The switch 2025 receives the decoded picture 2035 ol the object and a signal 2037 which is separately input as 
the background, and refers to the alpha-map signal 2022. If the interest pixel is the object, the switch 2025 selectively 
outputs the decoded picture 2035 of the object as a reconstructed picture 2036 to the outside. It the interest pixel is 
the background, the switch 2025 selectivery outputs the signal 2037 as the reconstructed picture 2036 to the outside. 
20 in the decoding apparatus with the above configuration, the alpha-code 2019 and the transform coefficient 2018 

are externally input. The alpha-code 201 9 is supplied to the decoder 2020, and the transform coefficient 201 8 is supplied 
to the inverse DCT circuit 2021 

The decoder 2020 decodes the alpha-map signal 2022 and outputs the decoded signal to the switch 2023, the 
decision circuit 2024, and the switch 2025. The inverse DCT circuit 2021 decodes the picture and supplies the decoded 
25 picture 2026 to the switch 2023. 

On the basis of the alpha-map signal 2022 decoded by the decoder 2020, the switch 2023 divides the decoded 
picture 2026 into the object picture 2027 and the background picture 2028 and discards the background picture 2028. 
: ' tHe object picture : 2027 separated by the switch 2023 is supplied to the switch 2029, the exfyander 2030, and the 
average circuit 2031 . 

30 M The average circuit 2031 calculates the average value 2032 of the object picture 2027 and supplies the average 
value 2032ito the expander 2030. 

s The expander 2030 expands the amplitude of the object picture 2027 around the average value 2032 and supplies 
the expanded picture 2033 thus obtained to the switch 2029. 

The decision circuit 2024 decides whether an interest pixel in the object picture 2027 is in contact with the back- 
55 ground and supplies the decision result 2034 to the switch 2029. If the decision result 2034 indicates that the interest 
pixel is in contact with the background, the switch 2029 selectively outputs the expanded picture 21033 as the decoded 
picture 2035 of the object If the interest pixel is not in contact with the background, the switch 2029 selectively outputs 
the object picture 2027 as the decoded picture 2035 of the object. 

The output decoded picture 2035 of the object from the switch 2029 is supplied to the switch 2025. The switch 
40 20i25 refers to the alpha-map signal 2022 and, if the interest pixel which is a pixel currently being processed is the 
object; outputs the decoded picture 2035 of the object as the reconstructed picture 2036 to the outside. If the interest 
pixel is the background, the switch 2029 selectively outputs the signal 2037, which is separately input as the back- 
ground, as the reconstructed picture 2036. Note that a reconstructed signal of a background picture which is separately 
encoded or a predetermined pattern is used as the background signal 2037. 
45 The foregoing are examples of the processing of decreasing the step. 

The examples of the processing of decreasing the step have been described above, and a description will return 
to the subject of the second embodiment. 

As already described above, the arbitrary shape orthogonal transform circuit 101 refers to the alpha-map signal, 
checks the interest region of the picture, divides the interest region of the picture into square blocks each consisting 
so of N x N pixels, and orthogonally transforms each block to obtain N x N transform coefficients. 

For a block containing the boundary between the object and the background, a transform coefficient for the object 
and a transform coefficient for the background are calculated. These transform coefficients are supplied to adders 1 1 0 
and 1 1 1 for the : enhancement layer and the base layer, respectively. 

Upon receiving the transform coefficient, the adder 1 11 of the base layer calculates a prediction error signal between 
ss this transform coefficient and a motion compensation prediction value (BMC) which is converted into an orthogonal 
tiarislorm coefficient and supplied from the motion ^compensation prediction section 601 The adder 111 supplies the 
result to a quantizer 1 31 The quantizer 1 31 quantizes the prediction error signal in accordance with the quantization 
scale Q_scale supplied from the encoding controller 430 and sujsplies the quantized signal to the variable-length en- 
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coder 141 and a dequantizer 171. 

The variable-length encoder 141 performs variable-length encoding for the quantised prediction error signal. The 
variable-length encoder 141 also performs variable-length encoding for the side information such as the mode infor- 
mation containing the quantization scale information supplied from the encoding controller 430 and the motion vector 
supplied from the motion vector detector 510. 

These variable-length codes obtained by the variable-length encoder 141 are supplied to the multiplexer 155. The 
multiplexer 1 55 multiplexes these variable-length codes together with an alpha-code 55 which is encoded and supplied 
to the multiplexer 155. The multiplexer 155 outputs the multiplexed signal to the output buffer 161 

The output buffer 161 outputs the multiplexed signal as an encoded bit stream 21 to a transmission line or a storage 
medium and also feeds the capacity of the buffer back to the encoding controller 430. In accordance with this buffer 
capacity the encoding controller 430 generates the optimum quantization scale Q_scale. 

The quantized value of the prediction error signal supplied to the dequantizer 1 7 1 is dequantized by the dequantizer 
171. The adder 191 adds the dequantized value to the motion compensation prediction value (BMC), thereby calculating 
a reconstructed value in the transform coefficient domain. The reconstructed value is supplied to the motion compen- 
sation prediction section 601 . 

In the enhancement layer, a selector 300 performs selection on the basis of the value of an output (BQ) from the 
quantizer 1 31 in the base layer. That is, the selector 300 adaptively switches the output (EMC) from the motion com- 
pensation prediction section 600 in the enhancement layer and the output (BMC) from the motion compensation pre- 
diction section 601 in the base layer for each transform coefficient by using a method to be described later and outputs 
the selected input as EP. 

More specifically, the output (BQ) from the quantizer 131 in the base layer is supplied to a binarizing circuit 310. 
If the value of BQ is "0", the binarizing circuit 310 outputs "0" to the selector 300. If the value is not "0", the binarizing 
circuit 310 outputs "1* to the selector 300. 

If the output from the binarizing circuit 31 0 is "0", the selector selectively outputs EMC as ER If the output is BMC, 
the selector 300 selectively outputs BMC as EP. Consequently, the transform coefficient output EMC from the motion 
compensation prediction section 600 in the enhancement layer is applied to a transform coefficient in a position where 
the output BQ from the quantizer 1 31 is B 0' : and the transform coefficient output BMC from the motion compensation 
prediction section 601 in the base layer is applied to a transform coefficient in a position where the output BQ from the ' 
quantizer 1 31 is not B 0\ 

The quantizer 1 31 in the base layer receives and quantizes the output from the adder 111 The adder 1 1 1 receives 
the output from the arbitrary-shape orthogonal transform circuit 101 and the motion compensation prediction value 
obtained from a picture ^ frame by the motion compensation prediction section 601 and 

calculates the difference between them. Therefore, if the motion compensation prediction value is correct, the difference 
between the two values output from the adder 11 1 is "0". 

Accordingly, of the quantized values as the output BQ from the quantizer 1 31 in the base layer, coefficients whose 
values are not "0" are coefficients representing that the motion compensation prediction is incorrect. 

If the motion compensation prediction section 600 performs motion compensation prediction by using the same 
motion vector as in the base layer supplied from the motion vector detector 51 0, it is estimated that motion compensation 
prediction for coefficients in the enhancement layer in the same positions as in the base layer is incorrect. 

For these coefficients, therefore, the selector 300 selects BMC as the output from the motion compensation pre- 
diction section 601 for the base layer. 

On the other hand, since it is estimated that motion compensation for other coefficients is correct, the selector 300 
selects the prediction value in the enhancement layer with a smaller encoding distortion. Consequently, the signal EC 
encoded in the enhancement layer is the quantized error signal of the base layer if motion compensation prediction is 
incorrect, and is the motion compensation prediction error signal of the enhancement layer if motion compensation 
prediction is correct. This improves the efficiency of encoding for coefficients the motion compensation prediction for 
which is incorrect. 

The adder 110 in the enhancement layer calculates a prediction error signal between the transform coefficient of 
the input picture supplied from the arbitrary-shape orthogonal transform circuit 1 01 and the output (EP) from the selector 
300 and supplies the result to an adder 121 . 

The adder 121 receives a dequantized value 30 of BQ supplied from the dequantizer 171 . Accordingly, the adder 
121 calculates the difference EC between the value 30 and the output from the adder 110 and supplies the difference 
EC as a prediction error signal to a quantizer 1 30. 

The quantizer 130 quantizes the signal EC by using the quantization scale Q_scate supplied from the encoding 
controller 420 in accordance with the buffer capacity. The quantizer 1 30 supplies the quantized output to a variable- 
length encoder 140 and a dequantizer 170. The variable-length encoder 140 separately performs variable-length en- 
coding for the quantized prediction error signal and the side information such as the mode information supplied from 
the encoding controller 420 and supplies the variable-length codes to a multiplexer 150. 
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The multiplexer 150 multiplexes these variable-length codes and supplies the multiplexed signal to the output 
buffer 160 The output buffer 1 60 temporarily holds the signal and outputs the signal as an encoded bit stream 20 to 
a transmission line or a storage medium. Also the output bufler 160 feeds the capacity of the buffer to the encoding 
controller 420 Upon receiving the buffer capacity, the encoding controller 420 generates the optimum quantization 
5 scale QJ scale corresponding to the capacity and supplies the quantization scale Q. scale to the quantizer 130 and the 
variable-length encoder 140. 

The quantized value supplied from the quantizer 130 to the dequantizer 170 is dequantized. Ah adder 1 80 adds 
the decjuantizecJ value to the output 30 supplied from the dequantizer 171 in the base layer, thereby reconstructing the 
prediction error signal. 

10 the adder 190 adds the prediction error signal reconstructed by the adder 180 to the motion compensation pre- 

diction value (EP) supplied I from the selector 300, calculating a reconstructed value in the transform coefficient domain. 
The reconstructed value is supplied to the motion compensation prediction section 600. 

FIG. 6 A is a block diagram of the motion compensation prediction sections 600 and 601. Each of the motion 
compensation prediction sections 600 and 601 comprises an arbitrary-shape inverse orthogonal transform circuit 610, 

is a frame memory 620. a motion compensation circuit 630, and an arbitrary shape orthogonal transform circuit 640. The 
arbitrary shape inverse orthogonal transform circuit 61 0 inversely orthogonally transforms a reconstructed picture signal 
as an inputsighal in accordance with an alpharmap signal. The frame memory 620 temporarily holds the inversely 
orthogonaify transformed signal in units of f rames. The motion compensation circuit 630 receives the information of a 
motion vector, extracts a picture in a position indicated by the motion vector in units of frames, and supplies the extracted 

20 picture to the arbitrary -shape orthogonal transform circuit 640. The arbitrary-shape orthogonal transform circuit 640 
orthogonally transforms the supplied picture in accordance with the alpha-map signal. In other words, the arbitrary- 
shape c^ogonal transform circuit 640 orthogonally transforms the motion compensation prediction value of an arbi- 
trary, shape, thereby obtaining a motion compensation prediction value in a transform coefficient domain. 
In this configuration, a reconstructed value in a transform coefficient domain is supplied to the motion 

25 prediction sections 600 and 601 . The arbitrary -shape inverse orthogonal transform circuit 610 in this motion compen- 
sation prediction section inversely transforms: the reconstructed value into a reconstructed picture signal in accordance 
with the a^ha-map signal 50 which is separately supplied. The reconstructed picture signal is stored in the frame 
merrory 6^: „ :, ;, 

Of the reference pictures stored in the frame memory 620, a picture in a position indicated by the motion vector 
30 is extracted by the motion compensation circuit 630 in the motion compensation prediction section in units of blocks, 
and the extracted picture is supplied to the arbitrary shape orthogonal transform circuit 640 in the motion compensation 
prediction section. Upon receiving the blocks of the picture, the arbitrary shape orthogonal transfomi circuit 640 or- 
thogorally transform the picture blocks in accordance with the externally supplied alpha-map signal 50. thereby or- 
thogonally transfo^ the motion compensation prediction value of an arbitrary shape. Consequently, the arbitrary 
35 shape orthogonal transform circuit 640 can calculate and output the motion compensation prediction value in the trans- 
form coefficient domain. In a block containing the boundary between the object and the background, transform coef- 
ficients of both the object and the background are calculated. 

From the reconstructed value in the transform coefficient domain, the motion compensation prediction sections 
600 and 601 calculate the motion compensation prediction values EMC and BMC in the transform coefficient domain 
40 and supplies the values to the selector 300, 

The foregoing is the explanation of the encoding apparatus of the second embodiment. The decoding apparatus 
of the second embodiment will be described below. 

FIG. 7 is a block diagram of the decoding apparatus of the present invention. 

This configuration differs from FIG 4 in three points; that is, a motion compensation prediction section 602 is 
45 provided instead of the motion compensation prediction section 201, a motion compensation prediction section 603 is 
provided iistead of the motion compensation prediction section 203, and a demultiplexer 157 is provided instead of 
the demultiplexer 163. 

Each of the motion compensation prediction sections 602 and 603 performs motion compensation prediction by 
referring to an alpha-map signal. The demultiplexer 153 demultiplexes a quantized value of a transform coefficient, a 
so motion vector, and side information such as a quantization scale and transfers the demultiplexed signals to the variable- 
length decoder 1 43 The demultiplexer 1 57 adcfitionaSy has a function of demultiplexing an alpha-code and transferring 
the demultiplexed codes to an alpha-map decoding apparatus (not shown). 

In this configuration, a base layer bit stream 23 which is formed by encoding and rnuftiplaxirtg a quantized value 
of a transform coefficient, a motion vector, side information such as a quantization scale, and an alpha-code is input 
ss to the input stage of the base layer. This bit stream 23 is stored in an input buffer 167 and supplied to the demultiplexer 
" 157, 

" The demultiplexer 157 demultiplexes the bit stream into the quantized value crfthe transfonr^ coefficient Jhernolion 
vector, the side information, and the alpha-code Of these demultiplexed codes, the quantized value of the transform 
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coefficient, the motion vector, and the side information are supplied to a variable-length decoder 1 43 and decoded into 
signals of the quantized value of the transform coefficient, the motion vector and the quantization scale. Note that a 
code (alpha-code) 56 of an alpha-map signal is supplied to an alpha-map decoding apparatus (not shown) where the 
code is converted into the alpha-map signal, and the signal is supplied to the motion compensation prediction sections 
s 602 and 603 

Of the signals decoded by the variable-length decoder 143. the quantized value of the transform coefficient is 
dequantized by a dequantizer 173 and supplied to an adder 193. The adder 193 adds the dequantized transform 
coefficient and a motion compensation prediction value in a transform coefficient domain supplied from the motion 
compensation prediction section 603, thereby calculating a reconstructed value in the transform coefficient domain. 

io This reconstructed value is supplied to the motion compensation prediction section 603 and inversely transformed 

into a reconstructed picture signal by an arbitrary-shape inverse orthogonal transform circuit 61 0. The signal is output 
as an output reconstructed picture signal 41 and stored in a frame memory 620 (FIG. 6B). 

An enhancement layer bit stream 22 formed by encoding and multiplexing signals of a quantized value of a trans- 
form coefficient and side information such as a quantization scale is input to the input stage of the enhancement layer 

is The bit stream 22 is stored in an input buffer 162 and supplied to a demultiplexer 152. The demultiplexer 152 demul- 
tiplexes the bit stream into a code of the quantized value of the transform coefficient and a code ot the side information. 

The codes demultiplexed by the demultiplexer 152 are supplied to a variable-length decoder 142 and decoded 
into signals of the quantized value of the transform coefficient and the quantization scale. The quantized value of the 
transform coefficient is dequantized by a dequantizer 172 and supplied to an adder 181. The adder 181 adds this 

20 dequantized value to a dequantized value 31 supplied from the dequantizer 173 of the base layer and supplies the 
sum to an adder 1 92. 

The adder 192 adds the output from the adder 181 and the signal EP supplied from the selector 300 and thereby 
calculates a reconstructed value in a transform coefficient domain. This reconstructed value is supplied to the motion 
compensation prediction section 602 and inversely transformed into a reconstructed picture signal by an arbitrary- 
25 shape inverse orthogonal transform circuit 61 0 (FIG. 6B) provided in the motion compensation prediction section 602 
The reconstructed picture signal is output as an output reconstructed picture signal 40 and stored in a frame memory 
620 provided in the motion compensation prediction section 602. 

Of the reference pictures stored in the frame memory 620, a picture in a position indicated by the motion vector 
is extracted by a motion compensation circuit 630 (FIG. 6B) in the motion compensation prediction section in units of 

30 blocks, and the extracted picture is supplied to an arbitrary shape orthogonal transform circuit 640 in the motion com- 
pensation prediction section. The arbitrary-shape orthogonal transform circuit 640 orthogonally transforms the motion 
compensation prediction value of an arbitrary shape in accordance with an externally supplied alpha-map signal 50, 
thereby calculating and outputting a motion compensation prediction value in a transform coefficient domain. For a 
block containing the boundary between the object and the background, transform coefficients for both the object and 

35 the background are calculated. 

In this way, the motion compensation prediction sections 600 and 601 calculate the motion compensation prediction 
values EMC and BMC in the transform coefficient domain from the reconstructed value in the transform coefficient 
domain and supplies these values to the selector 300. 

One modification of the motion compensation prediction sections 600, 601 , 602, and 603 of the second embodiment 

40 wilt be described below with reference to FIGS. 8A and 8B. This modification is accomplished by expanding a back- 
ground prediction system (e.g., Miyamoto et al.: "Adaptive Predictive Coding System Using Background Prediction", 
PCSJ88.7-4, pp. 93-94, Watanabe et aL: "Adaptive Four-Difference - DCT Coding System", PCSJ88.8-2, pp. 1 1 7-118), 
which is conventionally used to improve the efficiency of encoding of a background region hidden by the movement of 
an object, so that the system is also usable to overlapping of objects. 

45 As shown in FIGS. 8A and 8B, the motion compensation prediction section comprises an arbitrary-shape inverse 

orthogonal transform circuit 610, an SW circuit 650, frame memories 621 , 622, and 623, an SW circuit 651 , a motion 
compensation circuit 630, and an arbitrary shape orthogonal transform circuit 640. 

The arbitrary-shape inverse orthogonal transform circuit 610 inversely transforms a reconstructed picture signal 
in accordance with an alpha-map signal. When a reconstructed value in a transform coefficient domain is input to the 

so motion compensation prediction section, this reconstructed value is supplied to the arbitrary-shape inverse orthogonal 
transform circuit 61 0 as one component of the motion compensation prediction section. The circuit 61 0 inversely trans- 
forms the reconstructed value into a reconstructed picture signal in accordance with an alpha-map signal. This alpha- 
map signal is supplied from an alpha-map decoding apparatus (not shown) provided in the decoding system. 

The reconstructed picture signal inversely transformed by the arbitrary-shape inverse orthogonal transform circuit 

ss 61 0 is stored in one of the frame memories 621 , 622, and 623. The SW circuit 650 selects one of the frame memories 
621 , 622, and 623 into which the signal is stored. For example, of the frame memories 621 , 622, and 623, the frame 
memories 621 and 622 are used to store object pictures, and the frame memory 623 is used to store background 
pictures. The object frame memories are prepared for two frames to separately hold pictures of two different objects 
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appearing in a frame. If three or more different objects exist, it is only necessary to prepare the number of object frame 
memories corresponding to the number of objects and allow the switch circuit 650 to select a corresponding frame 
memory. 

The SW circuit 650 can store the reconstructed picture signal from the arbitrary -shape inverse orthogonal transform 
5 circuit 610 in one of the frame memories 621: 622, and 623 in accordance with an alpha-map signal by opening or 
closing the swfich in accordance with the alpha-map signal. 

The SW circuit 651 opens or closes the switch in accordance with the alpha-map signal, thereby selecting one of 
the frame memories 621, 622! and 623 in accordance with the alpha-map signal and reading out the reconstructed 
picture signal stdred in that memory. Of the reconstructed picture signal (reference picture) read out from the frame 
10 memory 621 622, or 623 via the SW circuit 651 , the motion compensation circuit 630 extracts a picture in a position 
indicated by a motion vector in units of blocks and supplies the extracted picture to the arbitrary-shape orthogonal 
transform circuit 640. 

The arbitrary-shape orthogonal transform circuit 640 orthogonally transforms the reconstructed picture signal of 
the picture in the position indicated by the motion vector, which is read out from the frame memory 621 , 622, or 623 
is via the SW circuit 651, on the basis of the alpha-map signal, thereby orthogonally transforming a motion compensation 
prediction value of a picture of an arbitrary shape indicated by the alpha-map signal That is, the circuit 640 calculates 
and outputs the mptk^ compensation prediction value in the transform coefficient domain. 

In the configuration shown in FIGS. 8A and SB, it is assumed that the alpha-map supplied to the arbitrary-shape 
inverse oftrioo/snal transform circuit 610 can specify one of a plurality of objects and a background to which a pixel 
20 belongs 

In this configuration, a reconstructed value in a transform coefficient domain is supplied to the motion compensation 
prediction section. A portion of this reconstructed value, i.e., a picture of an arbitrary shape indicated by an alpha-map 
is inversely • orthogonally transformed into a reconstructed picture signal by the arbitrary-shape inverse orthogonal 
transform circuit 610. The reconstructed picture signal is stored in one of the frame memories 621, 622, and 623 for 
25 the objects anci the background by the S W circuit 650 in accordance with the alpha-map signal. 

These stored signals are sequentially selected and read out by the SW circuit 651 in accordance with the alpha- 
map signal and supplied to the motion compensation circuit 630 where a motion compensation prediction value is 
calculated. ■ 

As described above, to calculate the motion compensation prediction value in the motion compensation circuit 
30 630; the SW circuit 651 forms the motion compensation prediction value from the reference pictures stored in the frame 
memories. This improves the efficiency of encoding of a region which is hidden by overlapping of objects. 

The motion compensation prediction value -calculated by the motion compensation circuit 630 is supplied tp the 
arbitrary-shape orthogonal transform circuit 640 and orthogonally transformed on the basis of the alpha-map signal. 
The result is an orthogonal transform coefficient of the motion compensation prediction value of the picture of the 
35 arbitrary shape indicated by the alpha-map signal. 

In this modification as described above, to obtain the motion compensation prediction value in the motion com- 
pensation circuit 630, the motion compensation prediction value is formed for each of pictures stored in the frame 
memories which separately store pictures of the objects and the background in accordance with an alpha-map signal. 
This improves the efficiency of encoding of a region hidden by overlapping of the objects. 
40 Other configurations of the quantizers 130 and 131 and the dequantizers 170, 171, 172, and 173 in the first and 

second embodiments will be described below with reference to FIGS. 2 and 9 to 11. 

Quantization matrices shown in FIGS. 9 and 10 are described in TM5 as a test model of MPEG2. In each of FIGS. 
9 and 10, the matrix is represented by a two-dimensional matrix in a horizontal direction (h) and a vertical direction (v) 
with respect to 8 x 8 transform coefficients. 
45 The following equations show examples of quantization and dequantization using the quantization matrices in 

FIGS. 9 and 10. 
Quantization: 

^ level(v,h) = 

sign(coef(v,h))1coef(v.h)| *(v,h) 

/(2*Q_scate) (1) 

ss 

Inverse quantization: 
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coef'(v,h) - 



sing(level(v,h))(2* |level(v,h)| *e(v,h) 



/l6+i)*Q_scale 



where 



coef<v,h): 
levet(v.h): 
coef'(v.h): 
w(v h): 



Q_scate: 



transform coefficient 
quantized value 

transform coefficient (reconstructed value) 
quantization matrix 
quantization scale 



The modification is related to a quantization matrix for changing the weight of a quantization step size for each 
transform coefficient. In the SNR scalability, quantization in the enhancement layer is performed more finely than in 
the base layer. 

In the base layer, therefore the quantization matrices as shown in FIGS. 9 and 10 are used to finely quantize low- 
frequency transform coefficients and roughly quantize high-frequency coefficients. If this is the case, the subjective 
evaluation often improves when encoding is performed at the same encoding rate as when encoding is performed with 
a fixed quantization step size. Also, the coding efficiency is increased by increasing the occurrence probability of 0 by 
enlarging the center dead zone in a quantizer. This improves the quality of reconstructed pictures at low rates. 

In the enhancement layer, on the other hand, if high-frequency transform coefficients are roughly quantized, no 
fine textures are reconstructed to result in visual degradation. This also increases the influence of leedback quantization 
noise in high-frequency transform coefficients. 

Accordingly, in the enhancement layer a quantization matrix is used only for a transform coefficient whose quantized 
value BQ in the base layer is not "0V FIG. 11 shows an example of a quantization matrix obtained for the example 
shown in FIG. 2. When this matrix is used, the quantization error of a transform coefficient whose motion compensation 
prediction error is large is increased in the enhancement layer However, a quantization error in a largety changing 
portion is inconspicuous due to the masking effect of visual characteristics, and the resulting visual degradation is little. 

An example of transform to a one-dimensional sequence performed when a quantized transform coefficient is 
vanable-length-encoded in the first and second embodiments will be described below with reference to FIGS. 2, 12, 
and 13. This transform to a one-dimensional sequence is generally done by using a transform method called zigzag 
scan shown in FIG. 12. 

FIG. 12 shows a two-dimensional matrix divided into eight portions in each of a horizontal direction (h) and a 
vertical direction (v). In FIG! 12. 8 x 8 transform coefficients are arranged in increasing order of numbers given in 
measures. Consequently, low-frequency transform coefficients are arranged first and high-frequency transform coef- 
ficients are arranged next. Therefore, the larger the ordinal number for a quantized value, the higher the probability of 
*0", and this increases the coding efficiency when a combined event of the number of 0 runs and quantized values 
after the 0 runs is variable-length-encoded. This makes use of the properties that a lower-frequency coefficient has a 
higher electric power. 

In this example, therefore, the scan order in the enhancement layer is such that transform coefficients in positions 
where the quantized value BQ of the base layer is not "0* are arranged before transform coefficients in positions where 
BQ is "0" so that the transform coefficients are arranged in increasing order of zigzag scan numbers. 

That is, a transform coefficient in a position where BQ is "O" is a motion compensation prediction error signal in 
the enhancement layer, and a transform coefficient in a position where BQ is not *0* is a quantization error in the base 
layer. Accordingly, the above method ts based on the assumption that the statistical properties of the two transform 
coefficients are different. FIG. 13 shows a scan order corresponding to the example shown in FIG. 2. In FIG. 13, the 
order of numbers given in measures is the scan order. 

In the above example, it is assumed that transform bases do not overlap between blocks. On the other hand, 
reference: 'Ikuzawa et ai., Video Encoding Using Motion Compensation Filter Bank Structure, PCSJ92.8-5, 1992' has 
proposed an encoding method using a motion compensation fitter bank structure in which a decrease of the coding 
efficiency is little even when bases overlap each other because a transformed difference arrangement is used. 

The concept of the above reference is applicable to a prediction encoding apparatus (transformed difference ar- 
rangement) in an orthogonal transform coefficient domain as in the present invention. Therefore, the motion compen- 
sation filter bank structure can be applied to the above example. 

In the second embodiment described above, a frame of a video picture is orthogonally transformed by dividing the 
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frame into matrices each having a predetermined number (NxN)of pixels to obtain transform coefficients for individual 
bands of the spacial frequency, and motion compensation is performed in a transform coefficient domain for each of 
the N x N transform coefficients in each of upper and lower layers In this video encoding, motion compensation is 
performed for a picture in an interest region by using alpha-map information. When this motion compensation is per- 

5 formed, whether motion compensation prediction is correct is checked on the basis of an already decoded and quan- 
tized value in the tower layer (base layer) If the motion compensation prediction is correct encoding in the upper layer 
(enhancement layer) is performed by using a motion compensation prediction value obtained for the upper layer and 
having a smaller encoding distortion. If the motion compensation prediction is incorrect, encoding in the upper layer is 
done by using a motion compensation prediction value obtained for the lower layer (base layer) and having a larger 

10 encoding .distortion than that in the enhancement layer. This improves the coding efficiency for a coefficient the motion 
compensation prediction for which is incorrect and realizes an encoding system capable of encoding with little decrease 
in the coding efficiency. 

Accordingly, the resolution and the image quality can be varied in an arbitrary-shape picture encoding apparatus 
which separately encodes the background and the object. In addition, it is possible to provide scalable encoding and 
is decoding; apparatuses having a high coding efficiency. 

the third embodiment will be described below with reference to FIG. 14. 

As shown in FIG. 14, in blocks (enclosed by the solid tines) containing the boundary between the object and the 
background, motion vectors are separately detected for the object and the background. Since the number of pixels of 
either the object or the background decreases, the influence of noise increases, and this decreases the reliability of 
20 the motion vector. 

In blocks in the boundary, therefore, the motion vector detection range is made narrower than that in other blocks 
(indicated by the broken lines). 

Also, the object in a current frame has moved from the object in a reference frame. Therefore, erroneous detection 
of a motion vector can be reduced by limiting the motion vector search range for the object to the inside of the object 
2S in the reference frame. Limiting the search range also has an effect of decreasing the amount of motion vector search 
calculations Likewise, a motion vector is calculated from a background portion in the background. 

As described above, a large error can be prevented by making the motion vector detection range in blocks in the 
bouno^ narrower than that in other bk)cks (ir^icaled by the broken Itf^s). 

Finally^ as ah application of the present invention, an embodiment of a video transmission system to which the 
30 video encodhg and decoding apparatuses of the present invention are applied will be described below with reference 
toFIGS 15Ato15C 

In this system as shown in FIG. 15 A, an input video signal from a camera 1002 of a personal computer (PC) 1001 
is encoded by a Video encoding ap^ratus incorporated into the PC 1001. The encoded data output from the video 
encoding apparatus is multiplexed with other data of sounds or information. The mutiplexed data is transmitted by 
35 radio from a radio transceiver 1 003 and received by another radio transceiver 1 004. 

The signal received by the radio transceiver 1004 is demultiplexed into the encoded data of the video signal and 
the data of sounds or information; The encoded data of the video signal is decoded by a video decoding apparatus 
incorporated, into a workstation (EWS) 1005 and displayed on the display of the EWS 1005 

An input videb signal from a camera 1006 of the EWS 1005 is encoded in the same fashion as above by using a 
40 video encoding apparatus incorporated into the EWS 1 005. The encoded data of the video signal is multiplexed with 
other data of sounds or information. The multiplexed data is transmitted by radio from the radio transceiver 1004 and 
received by the radio transceiver 1003 The signal received by the radio transceiver 1003 is demultiplexed into the 
encoded data of the video signal and the data of sounds or information. The encoded data of the video signal is decoded 
by a video decoding apparatus incorporated into the PC 1001 and displayed on the display of the PC 1001 
45 FIG 15B is a block diagram schematically showing the video encoding apparatus incorporated into the PC 1001 

and the EWS 1005 in FIG. 15A. FIG. 15C is a block diagram schematically showing the video decoding apparatus 
incorporated into the PC 1001 and the EWS 1005 in FIG. 15A. 

The video encoding apparatus in FIG . 1 5B comprises an information source encoding section 1 1 02 which receives 
a picture signal from a video input section 1101 such as a camera and has an error resilience processor 1103, and a 
so transmission line encoding section 11 04. The information source encoding section 1101 performs discrete cosine trans- 
form (DCT) for a prediction error signal and quantizes the formed OCT coefficient The transmission tine encoding 
section 1104 performs variable-length encoding, error detection for encoded data, and error correcting coding. The 
encoded data output from the transmission line encoding section 1104 is supplied to a radio transceiver 1105 and 
transmitted. The processing in the data source encoding section 1101 and the variable -length encoding in the trans- 
£5 mission line encoding section 1 104 is done by apply ing processing methods such as explained in the embodiments of 
the present invention. 

The video decoding apparatus shown in FIG 15C comprises a transmission line decoding section 1202 and a 
data source decodffig section 1 203 having an error resilience processor 1 204 The transmission line decoding section 
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i 202 receives encoded data received by a radio transceiver 1 201 and performs processing which is the reverse of the 
processing done by the transmission line encoding section 1 1 04 for the input encoded data The data source decoding 
section 1203 receives the output signal from the transmission line decoding section 1202 and performs processing 
which is the reverse of the processing done by the data source encoding section 1102 for the input signal. The picture 
decoded by the data source decoding section 1 203 is output by a video output section 1 025 such as a display. 

The decoding processing in these sections is performed by applying processing methods such as explained in the 
embodiments of the present invention. As has been described above, the present invention accomplishes scalable 
encoding in which the quality of an arbitrary-shape picture can be varied step by step without largely decreasing the 
coding efficiency. Also, in the present invention, it is possible to decrease the amount of generated codes when OCT 
is performed for a picture of an arbitrary shape. 



Claims 

1 . A video encoding apparatus characterized by comprising: 

an orthogonal transform circuit (100) for orthogonally transforming an input picture signal to obtain a plurality 
of transform coefficients; 

a first local decoder (170, 180, 190, 200) for outputting first transform coefficients for a fine motion compen- 
sation prediction picture on the basis of a previous picture; 

a second local decoder (171 1 1 91 , 201 ) for outputting second transform coefficients for a coarse motion com- 
pensation prediction picture on the basis of a current picture corresponding to the input picture signal; 
means (310) for detecting a degree of motion compensation prediction. in said second local decoder; 
a selector (300) for selectively outputting the first and second transform coefficients in accordance with the 
degree of motion compensation prediction; 

a first calculator (110) for calculating a difference between the transform coefficients of said orthogonal trans- 
form circuit and ones of the first and second transform coefficients which are selected by said selector, and 
outputting a motion compensation prediction error signal; 

a first quantizer (130) for quantizing the motion compensation prediction error signal from said first adder and 
outputtinga first quantized motion compensation prediction error signal: 

a second calculator (111) for calculating a difference between the second transform coefficients from said 
second local -decoder and the transform coefficients from the orthogonal transform circuit, and outputting a 
second motion compensation prediction error signal; 

a second quantizer (131) for quantizing the motion compensation prediction error signal from said second 
calculator, and outputting a second quantized motion compensation prediction error signal; and 
an encoder (140. 141) for encoding the first and second quantized motion compensation prediction error sig- 
nals and outputting encoded signals. 

2. An apparatus according to claim 1 , characterized in that said orthogonal transform circuit (100) divides the input 
picture signal into a plurality of blocks each having a size of N x N pixels, orthogonally transforms the picture 
signal in units of blocks each having the size of N x N pixels, and outputs N x N transform coefficients to said 
first and second adders in units of blocks. 

3. An apparatus according to claim 1 f characterized in that said means (310) for detecting degree of motion com- 
pensation prediction compares the second transform coefficient from said second local decoder (171 , 191 , 201 ) 
with the transform coefficient from said orthogonal transform circuit (100) and outputs a detection signal corre- 
sponding to the degree of motion compensation prediction, and said selector (300) selects the first transform 
coefficient from said first local decoder in response to a detection signal representing that motion compensation 
prediction is correct, and selects the second transform coefficient from said second local decoder in response to 
a detection signal representing that motion compensation prediction is incorrect. 

4. An apparatus according to claim 1 , which comprises a motion vector detector (500) for detecting a motion vector 
from the input picture signal, said first local decoder comprises a first dequantizer (170) for dequantizing the first 
quantized motion compensation prediction error signal from said first quantizer and outputting a first output signal, 
a first adder (1 90) for adding the first output signal from the first dequantizer and the transform coefficient selected 
by said selector and outputting a first local decoded signal, and a first motion compensation prediction circuit (200) 
for obtaining the first transform coefficient on the basis of the first local decoded signal from said first adder and 
the motion vector, and said second local decoder comprises a second dequantizer (171) for dequantizing the 
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second quantized motion compensation prediction error signal from said second quantizer, and outputting a second 
output signal, a. second adder (191) for adding the second output signal from said second dequantizer and the 
second transform coefficient and outputting a second local decoded signal, and a second motion compensation 
prediction circuit (201) for obtaining the second transform coefficient on the basis of the second local decoded 
signal from said second adder and the motion vector. 

A video encoding apparatus characterized by comprising: 

an orthogonal transform circuit (1 00) for dividing an input video signal into a plurality of blocks each containing 
N x N pixels and orthogonally transforming the input video signal in units of blocks to obtain a plurality of 
transform coefficients divided in spacial frequency bands; 

a first motion prediction processing section (130, 170, 180, 190, 200) for performing motion compensation 
prediction processing for the plurality of transform coefficients in order to obtain an upper-layer motion com- 
pensation prediction signal having the number of data enough to obtain a high image quality; 
a second motion prediction processing section (131. 171, 191,201) for performing motion compensation pre- 
diction processing for the plurality of transf orm coefficients in order to obtain a lower-layer motion compensation 
prediction signal upon reducing the number of data; 

a decision section (310) for deciding in motion compensat ion on the basis of the lower-layer motion compen- 
sation prediction signal whether motion compensation prediction is correct; 

a selector (300) for selecting the upper-layer motion compensation prediction signal in response to a decision 
representing a correct motion compensation prediction from said decision section, and the lower-layer motion 
compensation prediction signal in response to a decision representing an incorrect motion compensation pre- 
diction; and 

an encoder (140, 141) for encoding one of the upper-layer motion compensation prediction signal and the 
lower-layer motion compensation prediction signal which is selected by said selector. 

A video encoding apparatus for realizing SNR scalability in M layers, characterized by comprising: 

an orthogonal transform circuit (100) for dividing an input video signal intoa plurality of blocks each containing 
N x N pixels and orthogonally transforming the input video signal in units of blocks to obtain a plurality of 
transform coefficients divided in spacial frequency bands; 

a first mot^ prediction processing section (170, 180, 190, 200) for performing motion com- 

pensation prediction processing for the plurality of transform coefficients in order to obtain an mth-layer (m = 
2 to M) motion compensation prediction signal; 

a second motion compensation prediction processing section (171, 181, 191, 1 21 }for performing nrwtion com- 
pensation prediction processing for the plurality of transform coefficients in order to obtain an (m-1 )th-layer 
motion compensation prediction signal; 

switching means (300) for selecting the mth-layer motion compensation prediction signal of said first motion 
compensation prediction processing section in order to obtain an mth-layer prediction value when a quantized 
output from said second motion compensation prediction processing section is 0, and switching between the 
mth-layer motic>nc^ signal and the (m-1 )th-layer motion compensation prediction signal 

in units of transform coefficients in order to select the (m-1 )th-layer motion compensation prediction signal 
when the quantized output is not less than 1 ; 

means (120) for calculating a difference signal between an (m-1 )th-layer dequantized output from said second 
motion compensation prediction processing section and an mth-layer motion compensation prediction error 
signal obtained by a difference between the mth-layer motion compensation prediction signal and the transform 
coefficient from said orthogonal transform circuit; and 

encoding means ( 1 30, 1 40) for quantizing and encoding the difference signal to output an encoded bit stream. 

An apparatus according to claim 6, characterized in that said switching means comprises means (300) for switching 
an mth-layer (m = 2 to M) quantization matrix between a transform coefficient corresponcfing to a quantized output 
of zero in the (m-1 )th layer and a transform coefficient corresponding to a quantized output of not less than 1 . 

An apparatus according to claim 6, characterized by further comprising means for, when quantized values of two- 
dimensional orthogonal transform coefficients in the mth layer (m = 2 to M) are to be arranged into a one-dimen- 
sional series, arranging a transform coefficient con-esponding to a quantized output of not less than 1 in the (nrv 
1 )th layer and then a transform coefficient corresponding to a quantized output of 0 in the (m-1 )th layer in an order 
named. 
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9. An apparatus according to claim 6, characterized in that said orthogonal transform circuit comprises means (101) 
for transforming an arbitrary- shape picture into a transform coefficient in accordance with an alpha-map signal for 
discriminating a background of an input picture from an object thereof, said encoding means comprises means for 
encoding the alpha-map signal, said first motion compensation prediction processing section comprises means 
for inversely transforming the transform coefficient in accordance with the encoded alpha-map signal to reconstruct 
the arbitrary-shape picture. 

10. .A video encoding/decoding system characterized by comprising: 

a video encoding apparatus for realizing SNR (Signal to Noise Ratio) scalability in M layers, which includes, 

an orthogonal transform circuit (100) for dividing an input video signal into a plurality of blocks each con- 
taining N x N pixels and orthogonally transforming the input video signal in units of blocks to obtain a 
plurality of transform coefficients divided in spacial frequency bands, 

a first motion compensation prediction processing section (170, 180, 190, 200) for performing motion 
compensation prediction processing for the plurality of transform coefficients in order to obtain an mth- 
layer (m = 2 to M) motion compensation prediction signal, 

a second motion compensation prediction processing section (171 ,191, 201) for performing motion com- 
pensation prediction processing for the plurality of transform coefficients in order to obtain an (m-i)th- 
layer motion compensation prediction signal, 

switching means (300) for selecting the mth-layer motion compensation prediction signal of said first mo- 
tion compensation prediction processing section in order to obtain an mth-layer prediction value when a 
quantized output from said second motion compensation prediction processing section is 0, and switching 
between the mth-layer motion compensation prediction signal and the (m-l)th-layer motion compensation 
prediction signal in units of transform coefficients in order to select the (m-1 )th-layer motion compensation 
prediction signal when the quantized output is not less than 1, 

means (110) for calculating a difference signal between an (m-l)th-layer dequantized output from said 
second motion compensation prediction processing section and an mth-layer motion compensation pre- 
diction error signal obtained by a difference between the mth-layer motion compensation prediction signal 
and the transform coefficient from said orthogonal transform circuit, and 

encoding means (130, 140) for quantizing and encoding the difference signal to output an encoded bit 
stream; ano* . 

a video decoding apparatus which includes, 

means (152 : 1 53) for extracting codes up to a code in the mth (m = 2 to M) layer from the encoded bit 
stream from said video encoding apparatus, 

decoding means (142, 143) for decoding the codes of respective layers up to the mth layer, 
dequantization means (1 72, 1 73) for dequantizing, in the respective layers, the quantized values decoded 
by said decoding means, 

switching means (300) for switching the mth-layer (m = 2 to M) motion compensation prediction value and 
the (m-1)th-iayer motion compensation prediction value in units of transform coefficients, and outputting 
the mth-layer motion compensation prediction value for the quantized output of 0 in the (m-1)th layer and 
the (m-1 )th-layer motion compensation prediction value for the quantized output of not less than 1 in the 
(m-1 )th layer in units of transform coefficients in order to obtain the mth-layer prediction value, and 
means (1 92) for adding the mth-layer motion compensation prediction value and the (m-1 )th-layer motion 
compensation prediction value to reconstruct the mth-layer motion compensation prediction error signal. 

11. A system according to claim 10, characterized by further comprising switching means for switching an mth-layer 
(m = 2 to M) quantization matrix between a transform coefficient corresponding to a quantized output of zero in 
the (m-1 )th layer and a transform coefficient corresponding to a quantized output of not less than 1 . 

12. A system according to claim 10, characterized in that said decoding means (142, 143) comprises means for de- 
coding an alpha-map signal for discriminating a background of an input picture from an object thereof, and means 
for inversely transforming the arbitrary-shape picture transformed into the transform coefficient in accordance with 
the encoded alpha-map signal to reconstruct the arbitrary-shape picture. 

1 3. A video encoding apparatus characterized by comprising: 
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an orthogonal transform circuit (101) for dividing an input video signal into a plurality of blocks each containing 
N X N pixels and orthogonally transforming an arbitrary-shape picture in units of blocks in accordance with 
an alpha-map /signal for discriminating a background of an input picture from at least one object thereof to 
obtain a plurality of transform coefficients; 

a first motion compensation prediction processing section (130, 170, 180. 190. 200) having a frame memory 
(620) corresponding to the background and at least one object, means for inversely transforming the transform 
coefficient in accordance with the alpha-map signal to reconstruct the arbitrary-shape picture, and means for 
performing motion compensation prediction processing for the plurality of transform coefficients, said first mo- 
tion compensation prediction processing section being adapted to obtain an upper-layer motion compensation 
prediction signal having the number of data enough to obtain a high image quality; 

a second motiOT compensatkxipredictiOT processing section (131 ( ^ 181. 191, 601) having a frame memory 
(620) corresponding to the background and at least one object, means for inversely transforming the transform 
coefficient in accordance with the alpha-map signal to reconstruct the arbitrary -shape picture, and means for 
performing motion compensation prediction processing for the plurality of transform coefficients, said second 
motion compensation prediction processing section being adapted to obtain an lower-layer motion compen- 
sation prediction signal upon reducing, the number of data; 

a decision section (31 0) for deciding in motion compensation on the basis of the lower motion compensation 
prediction signal whether motion compensation is correct; 

a selector (300) for selecting the upper-layer motion compensation prediction signal in response to a correct 
motion compensation prediction from said decision section, and the lower-layer motion compensation predic- 
tion signal in response to an incorrect motion compensation prediction; and 
an encoder (140) for encoding a motion compensation prediction value selected by said selector 

14. An apparatus according to claim 1 3, which includes means (510) for detecting a motion vector for each object and 
the background in a block containing a plurality of objects or the background, and a motion vector detection range 
in the block containing the plurality of objects or the background is narrower than that in a block containing one 
object or the background. 

15/ An apparatus according to clam 13, characterized by further comprising motion vector detection means (1 41 ) for 
detecting a motion vector in units of objects and performing motion vector search operation in the same object 

16. A vidro encoding apparatus characterized by comprising: 

ah orthogonal transform circuit (101) for dividing an input video signal into a plurality of btocks each containing 
N X N pixels and orthogonally transforming an arbitrary-shape picture in units of blocks to obtain a plurality 
of transform coefficients: 

means for encoding and outputting an alpha-map signal for discreninating a background of a picture from an 
object thereof; 

means (201 1 ) for calculating an average value of pixel values of an object portion using the alpha-map signal 
in units of blocks; 

means (2005) for assigning the average value to a background portion of the block; 

means (2004) for deciding using the alpha-map signal whether a pixel in the object is close to the background; 
means (2010) for compressing, about the average value, the pixel in the object decided to be close to the 
background; and 

means (201 7) for orthogonally transforming each block to output an orthogonal transform coefficient. 

17. A video encoding/decoding system characterized by comprising: 

a video encoding which includes, 

an orthogonal transform circuit (1 01 ) for dividing an input video signal into a plurality of btocks each con- 
sisting ofNxN pixels and orthogonally transforming an arbitrary-shape picture in units of blocks to obtain 
a plurality of transform coefficients, means for encoding and outputting an alpha-map signal for discrimi- 
nating a background of a picture from an object thereof, 

means (2011) for calculating an average value of pixel values of an object portion using the alpha-map 
signal in units of blocks, 

means (2005) for assigning the average value to a background portion of the block. 

means (2004) for deciding using the alpha-map signal whether a pixel in the object is close to the back- 
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ground. 

means (201 4) for compressing, about the average value, the pixel in the object decided to be ctose to the 
background, and 

means (201 7) for orthogonally transforming each block to output an orthogonal transform coefficient: and 

5 

a video decoding apparatus which includes, 

means (2020) for decoding the alpha-map signal encoded by said video encoding apparatus, 
means (2021 ) for decoding a block from the orthogonal translorm coefficient. 
J0 means (2031 ) for calculating an average value of the object portion, the background portion, or the entire 

block, 

means (2024) for deciding using the alpha-map signal whether the pixel in the object is close to the back- 
ground, and 

means (2030) for decompressing, about the average value, the pixel in the object decided to be close to 
is the background by said deciding means. 

18. A video encoding apparatus characterized by comprising: 

an orthogonal transform circuit (101 ) for dividing an input video signal into a plurality of blocks each containing 
20 N x N pixels and orthogonally transforming an arbitrary-shape picture in units of blocks to obtain a plurality 

of transform coefficients; 

means (2006) for encoding and outputting an alpha-map signal for discriminating a background of a picture 
from an object thereof; 

means (2011 ) for calculating an average value of an object portion in units of blocks for the picture, using the 
25 alpha-map signal: 

means (2005) for assigning the average value obtained by said means for calculating the average value to a 
background portion of the block: 

means (2004) for deciding using the alpha-map signal whether a pixel in the background is close to the object; 
means (2009) for correcting the pixel in the background which is decided to be close to the object as a decision 
30 result of said deciding means in such a manner that a pixel value of the pixel comes close to a pixel value of 

the object near the pixel; and 

means (2017) for outputting an orthogonal transform coefficient obtained by orthogonally transforming the 
block. . 
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